markparz (Fri, 03 Feb 2017 00:14:55 GMT):
Has joined the channel.
C0rWin (Fri, 03 Feb 2017 00:27:42 GMT):
@yacovm
yacovm (Fri, 03 Feb 2017 00:27:42 GMT):
Has joined the channel.
mastersingh24 (Fri, 03 Feb 2017 00:49:53 GMT):
Has joined the channel.
yacovm (Fri, 03 Feb 2017 00:51:25 GMT):
Ah?
mastersingh24 (Fri, 03 Feb 2017 00:56:09 GMT):
hello gents
salmanbaset (Fri, 03 Feb 2017 01:20:28 GMT):
Has joined the channel.
grapebaba (Fri, 03 Feb 2017 01:39:53 GMT):
Has joined the channel.
yacovm (Fri, 03 Feb 2017 01:40:24 GMT):
How was the hackathon @mastersingh24 ?
didnotgetagoodname (Fri, 03 Feb 2017 03:57:19 GMT):
Has joined the channel.
mastersingh24 (Fri, 03 Feb 2017 04:26:03 GMT):
hey - was getting on a flight earlier. it was pretty good. some good conversations on various subjects and looks like people are very interested in the new 1.0 architecture
mastersingh24 (Fri, 03 Feb 2017 04:26:05 GMT):
we are clearly going to need to get some good docs out with the alpha
C0rWin (Fri, 03 Feb 2017 07:49:09 GMT):
@mastersingh24 aloha
C0rWin (Fri, 03 Feb 2017 07:49:51 GMT):
You definitely need to make it to Israel, probably some hackathon with all startups we have here
JonathanLevi (Fri, 03 Feb 2017 07:58:56 GMT):
Has joined the channel.
cca88 (Fri, 03 Feb 2017 08:20:39 GMT):
Has joined the channel.
jdockter (Fri, 03 Feb 2017 11:38:26 GMT):
Has joined the channel.
silliman (Fri, 03 Feb 2017 13:56:30 GMT):
Has joined the channel.
AdnanC (Fri, 03 Feb 2017 14:24:27 GMT):
Has joined the channel.
gormand (Fri, 03 Feb 2017 14:42:47 GMT):
Has joined the channel.
mastersingh24 (Fri, 03 Feb 2017 14:48:21 GMT):
count me in!
kletkeman (Fri, 03 Feb 2017 15:19:28 GMT):
Has joined the channel.
danacr (Fri, 03 Feb 2017 15:23:53 GMT):
Has joined the channel.
Nishi (Fri, 03 Feb 2017 18:31:37 GMT):
Has joined the channel.
karkal (Fri, 03 Feb 2017 19:52:23 GMT):
Has joined the channel.
rickr (Fri, 03 Feb 2017 20:24:48 GMT):
Has joined the channel.
jiangyaoguo (Sat, 04 Feb 2017 01:31:10 GMT):
Has joined the channel.
David.Yan (Sat, 04 Feb 2017 03:29:48 GMT):
Has joined the channel.
genggjh (Sat, 04 Feb 2017 03:44:15 GMT):
Has joined the channel.
muralisr (Sat, 04 Feb 2017 14:46:46 GMT):
Has joined the channel.
muralisr (Sat, 04 Feb 2017 14:47:20 GMT):
me too please :-)
ray (Sun, 05 Feb 2017 08:42:19 GMT):
Has joined the channel.
mandler (Sun, 05 Feb 2017 10:47:17 GMT):
Has joined the channel.
rascal-3 (Sun, 05 Feb 2017 14:06:39 GMT):
Has joined the channel.
rascal-3 (Sun, 05 Feb 2017 14:09:13 GMT):
I need to know some of 1.0
yacovm (Sun, 05 Feb 2017 14:17:38 GMT):
hmm
yacovm (Sun, 05 Feb 2017 14:17:39 GMT):
?
rascal-3 (Sun, 05 Feb 2017 14:40:25 GMT):
ah
rascal-3 (Sun, 05 Feb 2017 14:40:36 GMT):
sorry, I will catch up!
Honglei (Sun, 05 Feb 2017 23:53:46 GMT):
Has joined the channel.
bryanhuang (Mon, 06 Feb 2017 04:32:56 GMT):
Has joined the channel.
jojocheung (Mon, 06 Feb 2017 07:32:12 GMT):
Has joined the channel.
gennadyl (Mon, 06 Feb 2017 08:19:52 GMT):
Has joined the channel.
david.peyronnin (Mon, 06 Feb 2017 09:48:35 GMT):
Has joined the channel.
harrijk (Mon, 06 Feb 2017 15:14:08 GMT):
Has joined the channel.
weeds (Mon, 06 Feb 2017 20:24:50 GMT):
Has joined the channel.
crazybit (Tue, 07 Feb 2017 05:31:43 GMT):
Has joined the channel.
zlliu (Tue, 07 Feb 2017 08:18:10 GMT):
Has joined the channel.
vitaly.ilinykh (Tue, 07 Feb 2017 08:41:02 GMT):
Has joined the channel.
raghavsood (Tue, 07 Feb 2017 08:49:02 GMT):
Has joined the channel.
mastersingh24 (Tue, 07 Feb 2017 12:57:50 GMT):
I'm here
mastersingh24 (Tue, 07 Feb 2017 12:58:04 GMT):
so you want me to start?
yacovm (Tue, 07 Feb 2017 12:58:24 GMT):
I am.
So, https://jira.hyperledger.org/browse/FAB-2007 deals with adding support for gossip to expose endpoints to peers according to whether they are in the same org as the source peer or not
yacovm (Tue, 07 Feb 2017 12:58:59 GMT):
I'm addressing this in the following way- in the `AliveMessage` I'm adding another field that's a remote endpoint.
yacovm (Tue, 07 Feb 2017 12:58:59 GMT):
I'm addressing this in the following way- in the `AliveMessage` I'm adding another field that's an internal endpoint.
yacovm (Tue, 07 Feb 2017 12:59:21 GMT):
to simplify things:
yacovm (Tue, 07 Feb 2017 13:00:25 GMT):
When an `AliveMessage` is propagated from peer to peer, once the `AliveMessage` has crossed to a peer that's in a different org, the inner endpoint is deleted.
yacovm (Tue, 07 Feb 2017 13:00:54 GMT):
now... a peer may start without a remote endpoint, just an inner endpoint
mastersingh24 (Tue, 07 Feb 2017 13:02:46 GMT):
ok - that makes sense - so then all we need in the peer config is some property called something like "advertiseAddress"
yacovm (Tue, 07 Feb 2017 13:03:03 GMT):
yeah. The idea I thought of is the following:
mastersingh24 (Tue, 07 Feb 2017 13:03:17 GMT):
since you can use "listenAddress" for the internal / default address
yacovm (Tue, 07 Feb 2017 13:03:19 GMT):
Now when you have 2 endpoints in the AliveMessage, you also have 2 signatures:
yacovm (Tue, 07 Feb 2017 13:03:28 GMT):
1 on the whole message, with the external endpoint which can be nil
yacovm (Tue, 07 Feb 2017 13:03:32 GMT):
and 1 on the inner endpoint
yacovm (Tue, 07 Feb 2017 13:04:12 GMT):
so if the messages "crosses" an organization boundry- the last peer that forwards it- deletes the inner endpoint, and doesn't need to re-sign the actual message which it can't since he may not be the source peer
yacovm (Tue, 07 Feb 2017 13:04:16 GMT):
but we have a problem
yacovm (Tue, 07 Feb 2017 13:04:32 GMT):
Let's say a peer `p` has been initialized without a remote endpoint and joins a channel
yacovm (Tue, 07 Feb 2017 13:04:38 GMT):
it contacts an anchor peer, tells it about itself
yacovm (Tue, 07 Feb 2017 13:05:17 GMT):
and now the anchor peer, which may be in a different org- publishes that peer's alive message with the inner endpoint in the "best" case and with *no endpoint* in the worst case
yacovm (Tue, 07 Feb 2017 13:05:33 GMT):
so what I want to do is something as the following which I think it's also a cool concept
yacovm (Tue, 07 Feb 2017 13:05:44 GMT):
peers that don't have external endpoints, are "invisible" to outside orgs
yacovm (Tue, 07 Feb 2017 13:05:54 GMT):
and may not join channels that are cross-orgs
yacovm (Tue, 07 Feb 2017 13:06:18 GMT):
I mean- the gossip part won't support cross-org channel messages for them
mastersingh24 (Tue, 07 Feb 2017 13:06:57 GMT):
would they still be able to connect and pull across orgs for say something like block/state transfer?
yacovm (Tue, 07 Feb 2017 13:07:09 GMT):
no
yacovm (Tue, 07 Feb 2017 13:07:26 GMT):
if they have no external endpoint, i don't think they should
yacovm (Tue, 07 Feb 2017 13:07:34 GMT):
if you want the peer to be able to do that, start it up correctly
yacovm (Tue, 07 Feb 2017 13:07:43 GMT):
you may even of course, have the same endpoint in both fields
yacovm (Tue, 07 Feb 2017 13:07:43 GMT):
you may even of course, have the same endpoint in both configs(internal and external)
mastersingh24 (Tue, 07 Feb 2017 13:11:01 GMT):
so basically, if a peer wants to participate in cross-org gossip (of course scoped to a channel), it must have an "advertiseAddress" in its config?
mastersingh24 (Tue, 07 Feb 2017 13:11:46 GMT):
of course, you could trick the system and populate this with a non-routable address as well and still be able to "connect" to someone else
yacovm (Tue, 07 Feb 2017 13:12:32 GMT):
and then no one will be able to connect to that peer
yacovm (Tue, 07 Feb 2017 13:12:44 GMT):
so why do that?
yacovm (Tue, 07 Feb 2017 13:13:02 GMT):
he could always get blocks from peers in its org that did publish an external address
mastersingh24 (Tue, 07 Feb 2017 13:13:30 GMT):
but that peer could still make an outbound connection to another peer and given that the GRPC connection is duplex, the peer it connected to could still send it messages and vice versa
yacovm (Tue, 07 Feb 2017 13:13:46 GMT):
yes I understand that
mastersingh24 (Tue, 07 Feb 2017 13:13:50 GMT):
agree - no reason to do it - but you could do it :)
yacovm (Tue, 07 Feb 2017 13:13:51 GMT):
but- I don't see the use case
yacovm (Tue, 07 Feb 2017 13:13:56 GMT):
oh yeah
yacovm (Tue, 07 Feb 2017 13:14:19 GMT):
but I also want users to be able not to config the peer/gossip in a way that causes problems that are hard to "debug"
yacovm (Tue, 07 Feb 2017 13:14:30 GMT):
Now, having said that- there is an alternative for this, however- and it is to create 2 membership overlays: 1 for inter-organization membership, and 1 for intra-organization membership.
The implementation is much simpler, but it of course induces "double" the messages in a certain sense.
mastersingh24 (Tue, 07 Feb 2017 13:14:31 GMT):
but I'm fine with saying that you need to specifically populate an address that will be used for gossip and if you don't have it, no cross-org gossip
mastersingh24 (Tue, 07 Feb 2017 13:15:01 GMT):
simplifies things a bit plus makes it clear what needs to happen to participate
yacovm (Tue, 07 Feb 2017 13:15:01 GMT):
I think I want to go with the 1st way at first, and if I'll see that it doesn't work well I might switch to the 2nd
mastersingh24 (Tue, 07 Feb 2017 13:15:10 GMT):
ok
yacovm (Tue, 07 Feb 2017 13:15:19 GMT):
so you're ok with that "invisible peer" idea?
mastersingh24 (Tue, 07 Feb 2017 13:16:48 GMT):
I think so - as I said, the only drawback is that an "invisible" peer cannot "pull", but I think it's ok to say "hey - if you are not willing to expose yourself, you can't play in cross-org gossip"
yacovm (Tue, 07 Feb 2017 13:16:58 GMT):
it can pull from its own org
mastersingh24 (Tue, 07 Feb 2017 13:17:05 GMT):
right
yacovm (Tue, 07 Feb 2017 13:17:16 GMT):
and it will most likely have an anchor peer to pull from, anyway
mastersingh24 (Tue, 07 Feb 2017 13:17:24 GMT):
good point
yacovm (Tue, 07 Feb 2017 13:18:51 GMT):
ok I just wanted to make sure we agree on the idea.
@C0rWin if you have some comments you're welcome
mastersingh24 (Tue, 07 Feb 2017 13:19:03 GMT):
so I suggest we explicitly add an item to the peer yaml config - maybe we call it "advertiseAddress" and put it under the "gossip" section?
yacovm (Tue, 07 Feb 2017 13:19:22 GMT):
*advertisedAddress*
mastersingh24 (Tue, 07 Feb 2017 13:19:30 GMT):
sounds good
mastersingh24 (Tue, 07 Feb 2017 13:19:41 GMT):
Kafka uses something like that as well
yacovm (Tue, 07 Feb 2017 13:19:57 GMT):
ok. Thanks Gari.
muralisr (Tue, 07 Feb 2017 13:25:03 GMT):
@yacovm so we will have two addresses advertisedAddress - for across org (but maybe different from anchor address, listenAddress - internal listen address) ?
yacovm (Tue, 07 Feb 2017 13:27:12 GMT):
So, now the gossip in the peer would advertise to peers in its own org both addresses, and to peers outside- only the external, and if the external address isn't specified, that peer won't be published to peers outside of the org.
yacovm (Tue, 07 Feb 2017 13:27:20 GMT):
The listenAddress has nothing to do with that
yacovm (Tue, 07 Feb 2017 13:27:33 GMT):
I'm not talking about https://github.com/hyperledger/fabric/blob/master/peer/core.yaml#L58-L64
muralisr (Tue, 07 Feb 2017 13:30:03 GMT):
ok. I saw `since you can use "listenAddress" for the internal / default address` from @mastersingh24 and wanted to make sure
muralisr (Tue, 07 Feb 2017 13:31:08 GMT):
so if there's an `advertisedAddress` a peer from across the org can communicate with that
muralisr (Tue, 07 Feb 2017 13:31:32 GMT):
and if not ?
C0rWin (Tue, 07 Feb 2017 13:37:33 GMT):
if peer won't have `advertiseAddress` it won't be able to communicate with other orgs (at least directly)
muralisr (Tue, 07 Feb 2017 13:39:32 GMT):
its not a "requirement" that we have advertisedAddress right ?
muralisr (Tue, 07 Feb 2017 13:40:44 GMT):
just trying to understand what we have to tell users to do
C0rWin (Tue, 07 Feb 2017 13:43:03 GMT):
not it's not, but if you'd like your peer to be able to communicate with external peers from different orgs, only then you will have to setup this parameter
muralisr (Tue, 07 Feb 2017 14:26:44 GMT):
sorry, let me play "dumb user" (likely a true description of me :-) )... in what circumstance should I specify `advertiseAddress`
yacovm (Tue, 07 Feb 2017 14:28:05 GMT):
it's `advertisedAddress`
yacovm (Tue, 07 Feb 2017 14:28:15 GMT):
and when you want your peer to connect to different orgs
mastersingh24 (Tue, 07 Feb 2017 14:30:33 GMT):
@muralisr - we actually played a similar trick in v0.6 with `peer.listenAddress` and `peer.address` for the discovery protocol. Basically discovery would advertise `peer.address` if present (for example this might be the expose IP:Port used by a Docker container whereas `listenAddress` was actually binding to localhost in the container which of course is not routable)
cbf (Tue, 07 Feb 2017 15:12:59 GMT):
Has joined the channel.
adc (Tue, 07 Feb 2017 15:18:00 GMT):
Has joined the channel.
mpage (Tue, 07 Feb 2017 15:31:28 GMT):
Has joined the channel.
troyronda (Tue, 07 Feb 2017 16:33:01 GMT):
Has joined the channel.
robear (Tue, 07 Feb 2017 16:34:44 GMT):
Has joined the channel.
AdnanC (Tue, 07 Feb 2017 20:10:47 GMT):
Hi @yacovm in the demo https://www.youtube.com/watch?v=Di4y1gauoKc , out of the first 10 peers, what as the leader setup? peer0 set to leader?
yacovm (Tue, 07 Feb 2017 20:20:16 GMT):
@AdnanC the demo is quite old, it was only for demonstrating development process.
We're in a much much more advanced stage of gossip and we have multi-channel support now, and also soon dynamic leader election.
Also notice that there wasn't any talk about channels in the demo, since it was made in a very early stage of V1.0 development.
As for the demo- there wasn't really a leader, we were using a peer that was disseminating blocks manually.
yacovm (Tue, 07 Feb 2017 20:21:09 GMT):
(by manually I mean- it was hardcoded to disseminate new blocks when it got them, and wasn't selected by some leader election)
AdnanC (Tue, 07 Feb 2017 20:21:23 GMT):
ok
yacovm (Tue, 07 Feb 2017 20:21:26 GMT):
Of course if you have questions, feel free to ask
AdnanC (Tue, 07 Feb 2017 20:21:45 GMT):
Thanks, are there any other recent demo/plabacks?
yacovm (Tue, 07 Feb 2017 20:24:41 GMT):
Hmm, I'm afraid that's the most up to date playback we have at the moment.
AdnanC (Tue, 07 Feb 2017 20:25:04 GMT):
ok, thanks. Will ask questions as you suggested :)
AdnanC (Tue, 07 Feb 2017 20:38:13 GMT):
About the changeset https://gerrit.hyperledger.org/r/#/c/5163/, is this the usecase?
"when the peers are are on different hostmachines, the endpoint will be overriden and set to the IP address of the respective machines so that Gossip data dissemination can take place."
yacovm (Tue, 07 Feb 2017 20:42:26 GMT):
of course.
yacovm (Tue, 07 Feb 2017 20:42:49 GMT):
for example if you're running your peer in docker and its local IP is 172.20.0.3
rahulhegde (Tue, 07 Feb 2017 20:44:20 GMT):
Has joined the channel.
beauson45 (Tue, 07 Feb 2017 20:57:08 GMT):
Has joined the channel.
yacovm (Tue, 07 Feb 2017 20:57:34 GMT):
you would want remote peers to connect to an ip address that is routable from them, @AdnanC
AdnanC (Tue, 07 Feb 2017 20:58:35 GMT):
right, ok.
mgk (Tue, 07 Feb 2017 22:08:41 GMT):
Has joined the channel.
xixuejia (Wed, 08 Feb 2017 01:39:21 GMT):
Has joined the channel.
ckeyer (Wed, 08 Feb 2017 02:07:19 GMT):
Has joined the channel.
johnm (Wed, 08 Feb 2017 02:27:03 GMT):
Has joined the channel.
niteshsolanki (Wed, 08 Feb 2017 08:55:04 GMT):
Has joined the channel.
MichalMalka (Wed, 08 Feb 2017 09:48:21 GMT):
Has joined the channel.
jvsteiner (Wed, 08 Feb 2017 11:01:50 GMT):
Has joined the channel.
Honglei (Wed, 08 Feb 2017 11:09:49 GMT):
@yacovm Hi, about the code-review, I just found that I created the chain with security-enabled. So, it's not a bug. I'll abandon that review. Thanks very much for the explanation.
yacovm (Wed, 08 Feb 2017 11:10:43 GMT):
of course, thank you for trying to patch our code!
patsToms (Wed, 08 Feb 2017 11:50:26 GMT):
Has joined the channel.
divyank (Wed, 08 Feb 2017 14:51:08 GMT):
Has joined the channel.
haidong (Wed, 08 Feb 2017 14:51:38 GMT):
Has joined the channel.
vbortnik (Wed, 08 Feb 2017 16:14:04 GMT):
Has joined the channel.
jkirke (Wed, 08 Feb 2017 16:33:33 GMT):
Has joined the channel.
StevenLanders (Wed, 08 Feb 2017 19:44:19 GMT):
Has joined the channel.
xiejunxi (Thu, 09 Feb 2017 03:28:12 GMT):
Has joined the channel.
chenxl (Thu, 09 Feb 2017 03:32:07 GMT):
Has joined the channel.
kenzhang (Fri, 10 Feb 2017 01:12:08 GMT):
Has joined the channel.
regorge (Fri, 10 Feb 2017 23:14:38 GMT):
Has joined the channel.
smithsj (Sat, 11 Feb 2017 08:49:33 GMT):
Has joined the channel.
yacovm (Mon, 13 Feb 2017 09:01:18 GMT):
Hi, I posted 3 JIRA tasks which are pretty easy to do, for anyone who wants to contribute to fabric code.
Any help will be greatly appreciated :)
https://jira.hyperledger.org/browse/FAB-2205
https://jira.hyperledger.org/browse/FAB-2206
https://jira.hyperledger.org/browse/FAB-2207
coolsvap (Mon, 13 Feb 2017 09:04:42 GMT):
Has joined the channel.
grapebaba (Mon, 13 Feb 2017 10:10:54 GMT):
:thumbsup:
pards (Mon, 13 Feb 2017 15:44:24 GMT):
Has joined the channel.
fz (Tue, 14 Feb 2017 00:03:32 GMT):
Has joined the channel.
joshhus (Tue, 14 Feb 2017 18:13:09 GMT):
Has joined the channel.
joshhus (Tue, 14 Feb 2017 18:13:52 GMT):
Fabric v1.0 gossip doc, proposed: https://gerrit.hyperledger.org/r/#/c/5717/
yacovm (Tue, 14 Feb 2017 18:15:22 GMT):
Thanks. we'll take a look and comment.
lehors (Tue, 14 Feb 2017 21:54:06 GMT):
Has joined the channel.
zwsyjj (Wed, 15 Feb 2017 02:15:05 GMT):
Has joined the channel.
jansony1 (Wed, 15 Feb 2017 17:26:34 GMT):
Has joined the channel.
joshhus (Wed, 15 Feb 2017 21:57:33 GMT):
Patch 2 comments responded to and incorporated in patch 3, thanks: https://gerrit.hyperledger.org/r/#/c/5717/3/docs/gossip.md
wutongtree (Thu, 16 Feb 2017 09:51:26 GMT):
Has joined the channel.
murrekatt (Thu, 16 Feb 2017 10:12:09 GMT):
Has joined the channel.
nickgaski (Thu, 16 Feb 2017 15:50:53 GMT):
Has joined the channel.
alviontaran (Thu, 16 Feb 2017 16:19:19 GMT):
Has joined the channel.
ylsGit (Fri, 17 Feb 2017 02:28:37 GMT):
Has joined the channel.
RajkumarNatarajan (Fri, 17 Feb 2017 03:56:08 GMT):
Has joined the channel.
xbee (Sat, 18 Feb 2017 10:42:51 GMT):
Has joined the channel.
dave.enyeart (Sat, 18 Feb 2017 12:15:57 GMT):
Has joined the channel.
dave.enyeart (Sat, 18 Feb 2017 12:21:38 GMT):
Hello gossipers, would it be possible to reduce the amount of gossip trace? The same gossip bytes fill the debug log every second over and over, even when there are no transactions go through the system, making it nearly impossible to find actual transactions in the debug. Here is one second of peer debug when nothing is happening:
yacovm (Sat, 18 Feb 2017 12:22:00 GMT):
Hey Dave
yacovm (Sat, 18 Feb 2017 12:22:07 GMT):
so, a question
yacovm (Sat, 18 Feb 2017 12:22:14 GMT):
are you using debug?
yacovm (Sat, 18 Feb 2017 12:22:17 GMT):
or info level?
yacovm (Sat, 18 Feb 2017 12:22:33 GMT):
the gossip inherits its logging levels from the "node" logging attribute
yacovm (Sat, 18 Feb 2017 12:22:43 GMT):
as the rest of the peer...
dave.enyeart (Sat, 18 Feb 2017 12:22:56 GMT):
```2017-02-18 11:58:22.532 UTC [gossip/comm#-1] Probe -> DEBU 8ee Entering, endpoint: 0.0.0.0:7051 PKIID: []
2017-02-18 11:58:22.532 UTC [gossip/comm#-1] Probe -> DEBU 8ef Returning
dave.enyeart (Sat, 18 Feb 2017 12:23:16 GMT):
actually it wouldnt let me paste it, it was too much
dave.enyeart (Sat, 18 Feb 2017 12:23:19 GMT):
peer debug
yacovm (Sat, 18 Feb 2017 12:23:27 GMT):
well- if the peer is in debug
yacovm (Sat, 18 Feb 2017 12:23:33 GMT):
what do you expect me to do?
dave.enyeart (Sat, 18 Feb 2017 12:23:37 GMT):
could you maybe print out the bytes the first time, and then only print them again if they change?
yacovm (Sat, 18 Feb 2017 12:23:52 GMT):
that's impossible logic-wise.
yacovm (Sat, 18 Feb 2017 12:24:01 GMT):
look at the sequence numbers
yacovm (Sat, 18 Feb 2017 12:24:03 GMT):
they increase
yacovm (Sat, 18 Feb 2017 12:24:13 GMT):
I'm trying to understand though
yacovm (Sat, 18 Feb 2017 12:24:29 GMT):
if you're after the DEBUG messages or the INFO one you pasted here?
dave.enyeart (Sat, 18 Feb 2017 12:24:30 GMT):
I see pages of gossip trace when nothing is happening in the system
yacovm (Sat, 18 Feb 2017 12:24:49 GMT):
would you agree to a situation
yacovm (Sat, 18 Feb 2017 12:24:59 GMT):
in which the gossip comm layer prints the Send() method in debug
yacovm (Sat, 18 Feb 2017 12:25:03 GMT):
instead of INFO as it is now?
yacovm (Sat, 18 Feb 2017 12:25:33 GMT):
because- what do you expect me to do with debug messages? we have debug messages as the rest of the system
dave.enyeart (Sat, 18 Feb 2017 12:25:36 GMT):
```2017-02-18 11:58:22.534 UTC [gossip/comm#-1] authenticateRemotePeer -> DEBU 8f4 Sending tag:EMPTY signature:"\030\001r\313\007\n \037\273\337\266\307\256[\312\266\303Le\301DR\221K3K\366x\205\271\032\355\320\226\211\237H\332\205\022\246\007\n\007DEFAULT\022\232\007-----BEGIN -----\nMIICjDCCAjKgAwIBAgIUBEVwsSx0TmqdbzNwleNBBzoIT0wwCgYIKoZIzj0EAwIw\nfzELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcTDVNh\nbiBGcmFuY2lzY28xHzAdBgNVBAoTFkludGVybmV0IFdpZGdldHMsIEluYy4xDDAK\nBgNVBAsTA1dXVzEUMBIGA1UEAxMLZXhhbXBsZS5jb20wHhcNMTYxMTExMTcwNzAw\nWhcNMTcxMTExMTcwNzAwWjBjMQswCQYDVQQGEwJVUzEXMBUGA1UECBMOTm9ydGgg\nQ2Fyb2xpbmExEDAOBgNVBAcTB1JhbGVpZ2gxGzAZBgNVBAoTEkh5cGVybGVkZ2Vy\nIEZhYnJpYzEMMAoGA1UECxMDQ09QMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE\nHBuKsAO43hs4JGpFfiGMkB/xsILTsOvmN2WmwpsPHZNL6w8HWe3xCPQtdG/XJJvZ\n+C756KEsUBM3yw5PTfku8qOBpzCBpDAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYw\nFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFOFC\ndcUZ4es3ltiCgAVDoyLfVpPIMB8GA1UdIwQYMBaAFBdnQj2qnoI/xMUdn1vDmdG1\nnEgQMCUGA1UdEQQeMByCCm15aG9zdC5jb22CDnd3dy5teWhvc3QuY29tMAoGCCqG\nSM49BAMCA0gAMEUCIDf9Hbl4xn3z4EwNKmilM9lX2Fq4jWpAaRVB97OmVEeyAiEA\n25aDPQHGGq2AvhKT0wvt08cX1GTGCIbfmuLpMwKQj38=\n-----END -----\n" conn:
yacovm (Sat, 18 Feb 2017 12:25:45 GMT):
yeah, that's a debug message though
dave.enyeart (Sat, 18 Feb 2017 12:25:58 GMT):
definitely shouldnt be info
yacovm (Sat, 18 Feb 2017 12:26:03 GMT):
it's not info
yacovm (Sat, 18 Feb 2017 12:26:05 GMT):
it's debug
dave.enyeart (Sat, 18 Feb 2017 12:26:17 GMT):
there is some INFO each second
dave.enyeart (Sat, 18 Feb 2017 12:26:22 GMT):
much more DEBUG
yacovm (Sat, 18 Feb 2017 12:26:26 GMT):
and I think that logging levels should be fine-grained per peer sub-component
yacovm (Sat, 18 Feb 2017 12:26:44 GMT):
the "some INFO" - unless I'm mistaken is only the Send() isn't it?
dave.enyeart (Sat, 18 Feb 2017 12:26:54 GMT):
right, but when there issue people turn on DEBUG for the entire peer and then they cant find the problem through the gossip gorp
dave.enyeart (Sat, 18 Feb 2017 12:27:34 GMT):
There are a few INFOs
dave.enyeart (Sat, 18 Feb 2017 12:27:39 GMT):
2017-02-18 11:58:22.536 UTC [gossip/comm#-1] GossipStream -> INFO 8fa Servicing 127.0.0.1:35198
yacovm (Sat, 18 Feb 2017 12:27:40 GMT):
yeah I agree here- but the solution I think is appropriate is to be able to control which sub-component of a peer needs its logging levels adjusted
yacovm (Sat, 18 Feb 2017 12:28:14 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=hi6MggrwJZYSSjM88)
I think it should stay INFO because it means a new peer connected to you
dave.enyeart (Sat, 18 Feb 2017 12:28:20 GMT):
2017-02-18 11:58:47.530 UTC [gossip/comm#-1] Send -> INFO 929 Entering, sending channel:"myc1" tag:CHAN_OR_ORG stateInfoPullReq:<> to 0 peers
yacovm (Sat, 18 Feb 2017 12:28:34 GMT):
yeah, that's the send() I was talking about
yacovm (Sat, 18 Feb 2017 12:28:37 GMT):
I can make it debug
yacovm (Sat, 18 Feb 2017 12:28:39 GMT):
if you'd like
dave.enyeart (Sat, 18 Feb 2017 12:29:14 GMT):
2017-02-18 12:17:33.100 UTC [gossip/comm#-1] authenticateRemotePeer -> DEBU 12ea Authenticated 127.0.0.1:7051
2017-02-18 12:17:33.101 UTC [gossip/comm#-1] createConnection -> DEBU 12eb Exiting
2017-02-18 12:17:33.101 UTC [gossip/comm#-1] sendToEndpoint -> DEBU 12ec Exiting
2017-02-18 12:17:33.101 UTC [gossip/comm#-1] authenticateRemotePeer -> DEBU 12ed Authenticated 127.0.0.1:35396
2017-02-18 12:17:33.101 UTC [gossip/comm#-1] GossipStream -> INFO 12ee Servicing 127.0.0.1:35396
2017-02-18 12:17:33.102 UTC [gossip/comm#-1] serviceConnection -> WARN 12ef Closing reading from stream
2017-02-18 12:17:33.102 UTC [gossip/comm#-1] writeToStream -> WARN 12f0 Closing writing to stream
2017-02-18 12:17:33.102 UTC [gossip/comm#-1] readFromStream -> WARN 12f1 [31 187 223 182 199 174 91 202 182 195 76 101 193 68 82 145 75 51 75 246 120 133 185 26 237 208 150 137 159 72 218 133] canceling read because closing
2017-02-18 12:17:33.102 UTC [gossip/comm#-1] readFromStream -> WARN 12f2 [31 187 223 182 199 174 91 202 182 195 76 101 193 68 82 145 75 51 75 246 120 133 185 26 237 208 150 137 159 72 218 133] Got error, aborting: EOF
2017-02-18 12:17:33.102 UTC [gossip/comm#-1] func2 -> INFO 12f3 Client 127.0.0.1:35396 disconnected
20
yacovm (Sat, 18 Feb 2017 12:29:19 GMT):
but as long as the gossip logging modules are inherited from the peer, I can't do anything about the debug messages
dave.enyeart (Sat, 18 Feb 2017 12:29:40 GMT):
WARNs also
yacovm (Sat, 18 Feb 2017 12:30:48 GMT):
hmmm- can you share the docker-compose file you used?
dave.enyeart (Sat, 18 Feb 2017 12:31:14 GMT):
using today's vagrant
dave.enyeart (Sat, 18 Feb 2017 12:31:22 GMT):
no docker
yacovm (Sat, 18 Feb 2017 12:31:31 GMT):
so the peer/core.yaml?
dave.enyeart (Sat, 18 Feb 2017 12:31:49 GMT):
using the default, no changes
yacovm (Sat, 18 Feb 2017 12:31:53 GMT):
aha I see
yacovm (Sat, 18 Feb 2017 12:31:54 GMT):
1 sec
yacovm (Sat, 18 Feb 2017 12:32:10 GMT):
https://github.com/hyperledger/fabric/blob/master/peer/core.yaml#L72
dave.enyeart (Sat, 18 Feb 2017 12:32:11 GMT):
but starting peer with CORE_LOGGING_LEVEL=DEBUG
yacovm (Sat, 18 Feb 2017 12:32:13 GMT):
remove this bootstrap line
yacovm (Sat, 18 Feb 2017 12:32:23 GMT):
and you won't see (I hope) the WARNings
dave.enyeart (Sat, 18 Feb 2017 12:33:08 GMT):
not all the gorp has sequence numbers.. for the stuff without sequence numbers can you print the gorp only the first time?
yacovm (Sat, 18 Feb 2017 12:33:20 GMT):
gorp?
dave.enyeart (Sat, 18 Feb 2017 12:33:27 GMT):
or remove sequence numbers from the gorp
dave.enyeart (Sat, 18 Feb 2017 12:33:41 GMT):
this is gorp:
dave.enyeart (Sat, 18 Feb 2017 12:33:43 GMT):
2017-02-18 12:20:28.157 UTC [gossip/comm#-1] authenticateRemotePeer -> DEBU 14a3 Received pkiID:"\037\273\337\266\307\256[\312\266\303Le\301DR\221K3K\366x\205\271\032\355\320\226\211\237H\332\205" cert:"\n\007DEFAULT\022\232\007-----BEGIN -----\nMIICjDCCAjKgAwIBAgIUBEVwsSx0TmqdbzNwleNBBzoIT0wwCgYIKoZIzj0EAwIw\nfzELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcTDVNh\nbiBGcmFuY2lzY28xHzAdBgNVBAoTFkludGVybmV0IFdpZGdldHMsIEluYy4xDDAK\nBgNVBAsTA1dXVzEUMBIGA1UEAxMLZXhhbXBsZS5jb20wHhcNMTYxMTExMTcwNzAw\nWhcNMTcxMTExMTcwNzAwWjBjMQswCQYDVQQGEwJVUzEXMBUGA1UECBMOTm9ydGgg\nQ2Fyb2xpbmExEDAOBgNVBAcTB1JhbGVpZ2gxGzAZBgNVBAoTEkh5cGVybGVkZ2Vy\nIEZhYnJpYzEMMAoGA1UECxMDQ09QMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE\nHBuKsAO43hs4JGpFfiGMkB/xsILTsOvmN2WmwpsPHZNL6w8HWe3xCPQtdG/XJJvZ\n+C756KEsUBM3yw5PTfku8qOBpzCBpDAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYw\nFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFOFC\ndcUZ4es3ltiCgAVDoyLfVpPIMB8GA1UdIwQYMBaAFBdnQj2qnoI/xMUdn1vDmdG1\nnEgQMCUGA1UdEQQeMByCCm15aG9zdC5jb22CDnd3dy5teWhvc3QuY29tMAoGCCqG\nSM49BAMCA0gAMEUCIDf9Hbl4xn3z4EwNKmilM9lX2Fq4jWpAaRVB97OmVEeyAiEA\n25aDPQHGGq2AvhKT0wvt08cX1GTGCIbfmuLpMwKQj38=\n-----END -----\n" from 127.0.0.1:35430
yacovm (Sat, 18 Feb 2017 12:34:29 GMT):
when a peer authenticates to a different peer, we print in DEBUG the message
dave.enyeart (Sat, 18 Feb 2017 12:34:30 GMT):
meaningless bytes to the administrator
yacovm (Sat, 18 Feb 2017 12:34:37 GMT):
why meaningless? look
yacovm (Sat, 18 Feb 2017 12:34:40 GMT):
you have the PEM here!
yacovm (Sat, 18 Feb 2017 12:34:48 GMT):
you can take this PEM
yacovm (Sat, 18 Feb 2017 12:34:52 GMT):
and understand who that peer was
dave.enyeart (Sat, 18 Feb 2017 12:34:53 GMT):
ok, just print it the first time
yacovm (Sat, 18 Feb 2017 12:34:57 GMT):
I can't
yacovm (Sat, 18 Feb 2017 12:35:08 GMT):
this is logically not possible
dave.enyeart (Sat, 18 Feb 2017 12:35:31 GMT):
ok, maybe we need multiple levels of debug then
yacovm (Sat, 18 Feb 2017 12:35:40 GMT):
I think what we really need is something else
yacovm (Sat, 18 Feb 2017 12:35:50 GMT):
the ability to define a fine-grained mapping
yacovm (Sat, 18 Feb 2017 12:35:58 GMT):
of which peer sub-component starts in which logging level
yacovm (Sat, 18 Feb 2017 12:36:12 GMT):
btw you can turn all these off with the peer logging control CLI did you know that?
yacovm (Sat, 18 Feb 2017 12:36:19 GMT):
(just making sure you did)
dave.enyeart (Sat, 18 Feb 2017 12:36:21 GMT):
i think there is component level debug, but when somebody doesnt know what the problem is they always turn on peer-wide debug
yacovm (Sat, 18 Feb 2017 12:36:37 GMT):
there isn't, and it doesn't really work
yacovm (Sat, 18 Feb 2017 12:36:53 GMT):
the reason it doesn't is that everyone in this project defines a logger with:
yacovm (Sat, 18 Feb 2017 12:37:09 GMT):
logging.getModule(mySubcomponent/sub-subcomponent)
yacovm (Sat, 18 Feb 2017 12:37:19 GMT):
therefore it's impossible to fine-tune these things
dave.enyeart (Sat, 18 Feb 2017 12:37:20 GMT):
ok, at least for me can you provide the instructions to enable all peer debug except gossip?
yacovm (Sat, 18 Feb 2017 12:37:29 GMT):
yeah sure :)
yacovm (Sat, 18 Feb 2017 12:37:32 GMT):
1 sec
dave.enyeart (Sat, 18 Feb 2017 12:38:03 GMT):
not to insult you, but i care about all debug except gossip :)
yacovm (Sat, 18 Feb 2017 12:38:14 GMT):
hehe, you're not
yacovm (Sat, 18 Feb 2017 12:38:49 GMT):
1 sec I'll build the peer
yacovm (Sat, 18 Feb 2017 12:39:33 GMT):
I think also I need to open a JIRA for that
yacovm (Sat, 18 Feb 2017 12:39:44 GMT):
maybe someone will take up the challange
yacovm (Sat, 18 Feb 2017 12:39:47 GMT):
and do this
yacovm (Sat, 18 Feb 2017 12:40:00 GMT):
this is an open source project after all
yacovm (Sat, 18 Feb 2017 12:40:18 GMT):
`./build/bin/peer logging`
yacovm (Sat, 18 Feb 2017 12:40:28 GMT):
you have there "setLevel"
yacovm (Sat, 18 Feb 2017 12:40:46 GMT):
so the gossip sub-component you need to silence is the thing in [ ]
dave.enyeart (Sat, 18 Feb 2017 12:41:25 GMT):
ok, i havent used that before, can you provide the statement i'd need?
yacovm (Sat, 18 Feb 2017 12:41:31 GMT):
yeah sure
yacovm (Sat, 18 Feb 2017 12:41:34 GMT):
let me try first before
yacovm (Sat, 18 Feb 2017 12:44:10 GMT):
```
yacovm@yacoVM ~/OBC/shared/gopath/src/github.com/hyperledger/fabric (master) $ ./build/bin/peer logging setlevel gossip/comm WARNING
2017-02-18 14:32:49.520 IST [loggingCmd] setLevel -> INFO 001 Log level set for peer module 'gossip/comm': WARNING
2017-02-18 14:32:49.520 IST [main] main -> INFO 002 Exiting.....
yacovm@yacoVM ~/OBC/shared/gopath/src/github.com/hyperledger/fabric (master) $
yacovm@yacoVM ~/OBC/shared/gopath/src/github.com/hyperledger/fabric (master) $
yacovm@yacoVM ~/OBC/shared/gopath/src/github.com/hyperledger/fabric (master) $ less out.log
yacovm@yacoVM ~/OBC/shared/gopath/src/github.com/hyperledger/fabric (master) $ ./build/bin/peer logging setlevel gossip/gossip WARNING
2017-02-18 14:33:02.852 IST [loggingCmd] setLevel -> INFO 001 Log level set for peer module 'gossip/gossip': WARNING
2017-02-18 14:33:02.852 IST [main] main -> INFO 002 Exiting.....
yacovm@yacoVM ~/OBC/shared/gopath/src/github.com/hyperledger/fabric (master) $ ./build/bin/peer logging setlevel gossip/discovery WARNING
2017-02-18 14:33:08.061 IST [loggingCmd] setLevel -> INFO 001 Log level set for peer module 'gossip/discovery': WARNING
```
yacovm (Sat, 18 Feb 2017 12:44:43 GMT):
so, basically if you see a log entry that has: `[gossip/subcomponent#identifier]`
yacovm (Sat, 18 Feb 2017 12:44:59 GMT):
you can do `peer logging setlevel gossip/subcomponent WARNING`
yacovm (Sat, 18 Feb 2017 12:45:05 GMT):
you can also set to error of course...
dave.enyeart (Sat, 18 Feb 2017 12:45:15 GMT):
great, thank you, will give it a try
yacovm (Sat, 18 Feb 2017 12:49:40 GMT):
no problem, and I also opened: https://jira.hyperledger.org/browse/FAB-2351
yacovm (Sat, 18 Feb 2017 12:49:43 GMT):
@dave.enyeart
dave.enyeart (Sat, 18 Feb 2017 12:53:05 GMT):
Nice and quiet now:
dave.enyeart (Sat, 18 Feb 2017 12:53:09 GMT):
peer logging setlevel gossip/comm#-1 WARNING
peer logging setlevel gossip/discovery#0.0.0.0:7051 WARNING
dave.enyeart (Sat, 18 Feb 2017 12:54:16 GMT):
please do silence the INFO and WARNING messages that still come out
yacovm (Sat, 18 Feb 2017 12:54:24 GMT):
ah so you need the thing after the # too?
dave.enyeart (Sat, 18 Feb 2017 12:54:27 GMT):
yes
yacovm (Sat, 18 Feb 2017 12:54:32 GMT):
I see
yacovm (Sat, 18 Feb 2017 12:54:41 GMT):
can you paste them?
dave.enyeart (Sat, 18 Feb 2017 12:54:56 GMT):
the INFO and WARNINGs I pasted above
dave.enyeart (Sat, 18 Feb 2017 12:55:19 GMT):
i'll get you a clean one...
yacovm (Sat, 18 Feb 2017 12:55:28 GMT):
I think the WARNING should go away if you delete the bootstrap config param
yacovm (Sat, 18 Feb 2017 12:55:35 GMT):
https://github.com/hyperledger/fabric/blob/master/peer/core.yaml#L72
dave.enyeart (Sat, 18 Feb 2017 12:57:53 GMT):
Here are the INFO and WARNINGs without the DEBUG:
dave.enyeart (Sat, 18 Feb 2017 12:58:00 GMT):
```2017-02-18 12:45:23.194 UTC [gossip/comm#-1] GossipStream -> INFO 459 Servicing 127.0.0.1:35522
2017-02-18 12:45:23.195 UTC [gossip/comm#-1] serviceConnection -> WARN 45a Closing reading from stream
2017-02-18 12:45:23.195 UTC [gossip/comm#-1] writeToStream -> WARN 45b Closing writing to stream
2017-02-18 12:45:23.195 UTC [gossip/comm#-1] readFromStream -> WARN 45c [31 187 223 182 199 174 91 202 182 195 76 101 193 68 82 145 75 51 75 246 120 133 185 26 237 208 150 137 159 72 218 133] canceling read because closing
2017-02-18 12:45:23.195 UTC [gossip/comm#-1] readFromStream -> WARN 45d [31 187 223 182 199 174 91 202 182 195 76 101 193 68 82 145 75 51 75 246 120 133 185 26 237 208 150 137 159 72 218 133] Got error, aborting: EOF
2017-02-18 12:45:23.195 UTC [gossip/comm#-1] func2 -> INFO 45e Client 127.0.0.1:35518 disconnected
2017-02-18 12:45:23.195 UTC [gossip/comm#-1] serviceConnection -> WARN 45f Closing reading from stream
2017-02-18 12:45:23.196 UTC [gossip/comm#-1] func2 -> INFO 460 Client 127.0.0.1:35522 disconnected
2017-02-18 12:45:23.196 UTC [gossip/comm#-1] readFromStream -> WARN 461 [31 187 223 182 199 174 91 202 182 195 76 101 193 68 82 145 75 51 75 246 120 133 185 26 237 208 150 137 159 72 218 133] canceling read because closing
2017-02-18 12:45:23.196 UTC [gossip/comm#-1] writeToStream -> WARN 462 Closing writing to stream
2017-02-18 12:45:23.196 UTC [gossip/comm#-1] serviceConnection -> WARN 463 Closing reading from stream
2017-02-18 12:45:23.196 UTC [gossip/comm#-1] writeToStream -> WARN 464 Closing writing to stream
2017-02-18 12:45:23.196 UTC [gossip/comm#-1] readFromStream -> WARN 465 [31 187 223 182 199 174 91 202 182 195 76 101 193 68 82 145 75 51 75 246 120 133 185 26 237 208 150 137 159 72 218 133] canceling read because closing
2017-02-18 12:45:23.196 UTC [gossip/comm#-1] writeToStream -> WARN 466 Closing writing to stream
2017-02-18 12:45:23.975 UTC [gossip/comm#-1] Send -> INFO 467 Entering, sending tag:EMPTY signature:"0D\002 S\006\354?\2457\226X\361&\360\311\217\302\236+o8-z-B\027p\276\260\231\021\371B\272t\002 /X\007\037\351\245\201\323\017h\r\205\234`Z\"3\\\234\345\321\357d\240\213\010\363\010\270\036\256F" aliveMsg:
dave.enyeart (Sat, 18 Feb 2017 12:58:35 GMT):
i'm ok now, just asking you remove those by default for others
yacovm (Sat, 18 Feb 2017 12:58:54 GMT):
wait I don't understand
yacovm (Sat, 18 Feb 2017 12:59:00 GMT):
have you managed to silence them or not?
dave.enyeart (Sat, 18 Feb 2017 12:59:48 GMT):
yes, i can silence them now. changed it back to INFO so that I could show you the INFOs and WARNINGs that come out. These should be DEBUG.
dave.enyeart (Sat, 18 Feb 2017 13:00:24 GMT):
when the system is at rest, the INFO/WARNING log should be empty
yacovm (Sat, 18 Feb 2017 13:00:25 GMT):
Hmm, yeah maybe you're right.
yacovm (Sat, 18 Feb 2017 13:00:45 GMT):
I mean about the DEBUG, not about the "when the system is at rest"
yacovm (Sat, 18 Feb 2017 13:00:51 GMT):
The system may be at rest to you
yacovm (Sat, 18 Feb 2017 13:00:56 GMT):
but gossip might do lots of things
yacovm (Sat, 18 Feb 2017 13:01:14 GMT):
therefore I think the optimal thing to do is to implement FAB-2351 I just opened
dave.enyeart (Sat, 18 Feb 2017 13:02:17 GMT):
when the system is at steady state and not processing any transactions, the INFO/WARNING log should be empty
dave.enyeart (Sat, 18 Feb 2017 13:03:27 GMT):
INFO is fine for initialization, but not steady state.
yacovm (Sat, 18 Feb 2017 13:10:32 GMT):
@dave.enyeart https://gerrit.hyperledger.org/r/#/c/6197/
dave.enyeart (Sat, 18 Feb 2017 13:12:37 GMT):
+1 Thanks!
yacovm (Sat, 18 Feb 2017 13:13:09 GMT):
np and tnx for bringing this up
warm3snow (Tue, 21 Feb 2017 05:18:31 GMT):
Has joined the channel.
jessilb (Tue, 21 Feb 2017 11:41:00 GMT):
Has joined the channel.
agaragiola (Tue, 21 Feb 2017 21:01:08 GMT):
Has joined the channel.
psa (Wed, 22 Feb 2017 15:55:49 GMT):
Has joined the channel.
v_thirugnanam (Sun, 26 Feb 2017 23:35:57 GMT):
Has joined the channel.
bkvellanki (Wed, 01 Mar 2017 20:15:20 GMT):
Has joined the channel.
Donald Liu (Thu, 02 Mar 2017 01:22:01 GMT):
Has joined the channel.
yoshtec (Fri, 03 Mar 2017 18:46:10 GMT):
Has joined the channel.
dave.enyeart (Tue, 07 Mar 2017 12:36:32 GMT):
@yacovm @C0rWin Looking at joinChain() resilience... @manish-sethi and I are trying to make the createLedger() call more resilient if there are any problems (e.g. cant connect to state database). Question - would it be possible to createLedger(), then createChain() and InitChain(), before committing the genesis block to ledger? So that gossip and everything can get initialized for the chain before the genesis block is committed. This way, if there is a problem committing the genesis block (e.g. cant connect to state database), then the genesis block could be gossipped later like any other block, after the problem gets resolved. Or must the genesis block be treated as a special case and therefore must get written before createChain() and InitChain()?
manish-sethi (Tue, 07 Mar 2017 12:36:32 GMT):
Has joined the channel.
C0rWin (Tue, 07 Mar 2017 12:38:00 GMT):
@dave.enyeart
C0rWin (Tue, 07 Mar 2017 12:38:00 GMT):
@dave.enyeart part of the gossip is the state transfer capabilities
C0rWin (Tue, 07 Mar 2017 12:39:06 GMT):
which assumes it starts with valid ledger which has at least one block (genesis) committed, so it could track the proper order of blocks comming
dave.enyeart (Tue, 07 Mar 2017 12:39:51 GMT):
Ok, I suspected as much, so the genesis block needs to get written prior to createChain() and initChain()
C0rWin (Tue, 07 Mar 2017 12:40:02 GMT):
currently yes
C0rWin (Tue, 07 Mar 2017 12:40:40 GMT):
while I can imagine how this could be changed/refactored so the state transfer layer would be initialized upon some event/callback from the ledger once it become ready
C0rWin (Tue, 07 Mar 2017 12:41:29 GMT):
also it seems, that there are many other parts will fail on the way before the gossip
dave.enyeart (Tue, 07 Mar 2017 12:41:45 GMT):
and what exactly does 'ready' mean? do you need some config from the genesis block? or could ready mean that the ledger infrastructure is in place, even if the genesis block is not there yet
dave.enyeart (Tue, 07 Mar 2017 12:42:21 GMT):
trying to see if genesis block itself could be gossipped later
C0rWin (Tue, 07 Mar 2017 12:42:25 GMT):
ready, ledger infrastructure is in place and block committed.
C0rWin (Tue, 07 Mar 2017 12:42:59 GMT):
I think this is doable
dave.enyeart (Tue, 07 Mar 2017 12:43:39 GMT):
but, the callback may not be until after the peer is restarted. it may require a restart to fix up the state database, etc
C0rWin (Tue, 07 Mar 2017 12:43:59 GMT):
wait
C0rWin (Tue, 07 Mar 2017 12:44:17 GMT):
of peer need to be restarted not sure what is the problem here?
C0rWin (Tue, 07 Mar 2017 12:44:45 GMT):
ledger infra is in place, you trying to commit GB, but failing, then restarting the peer to resolve the problem... Right?
C0rWin (Tue, 07 Mar 2017 12:45:08 GMT):
you will have to call join again and submit GB anyway, no?
C0rWin (Tue, 07 Mar 2017 12:45:27 GMT):
you need to initialize all MSP infrastructure using GB
C0rWin (Tue, 07 Mar 2017 12:45:53 GMT):
I'm just failing to see the exact flow you trying to solve currently
dave.enyeart (Tue, 07 Mar 2017 12:46:29 GMT):
with the current design we get into an inconsistent state, where ledger thinks it is created, but genesis block did not get committed yet. re-join will fail since you cannot re-create ledger
dave.enyeart (Tue, 07 Mar 2017 12:47:20 GMT):
we can do a larger re-design... but just wanted to check if it would be possible to 'catch up' the genesis block later via gossip, after the problem is resolved.
C0rWin (Tue, 07 Mar 2017 12:48:39 GMT):
if the invalid state of ledger could be propagated somehow to the gossip state layer
C0rWin (Tue, 07 Mar 2017 12:48:55 GMT):
then there should not be a problem I guess
C0rWin (Tue, 07 Mar 2017 12:49:04 GMT):
of course the evil in details
C0rWin (Tue, 07 Mar 2017 12:49:20 GMT):
but seems that it has to work
dave.enyeart (Tue, 07 Mar 2017 12:49:56 GMT):
let me ask this... does the gossip 'registration' get persisted somewhere? upon peer startup how do you know which channels to gossip on?
C0rWin (Tue, 07 Mar 2017 12:50:19 GMT):
we get this info from GB
C0rWin (Tue, 07 Mar 2017 12:50:58 GMT):
```// Initialize sets up any chains that the peer has from the persistence. This
// function should be called at the start up when the ledger and gossip
// ready
func Initialize(init func(string)) {
chainInitializer = init
var cb *common.Block
var ledger ledger.PeerLedger
ledgermgmt.Initialize()
ledgerIds, err := ledgermgmt.GetLedgerIDs()
if err != nil {
panic(fmt.Errorf("Error in initializing ledgermgmt: %s", err))
}
for _, cid := range ledgerIds {
peerLogger.Infof("Loading chain %s", cid)
if ledger, err = ledgermgmt.OpenLedger(cid); err != nil {
peerLogger.Warningf("Failed to load ledger %s(%s)", cid, err)
peerLogger.Debugf("Error while loading ledger %s with message %s. We continue to the next ledger rather than abort.", cid, err)
continue
}
if cb, err = getCurrConfigBlockFromLedger(ledger); err != nil {
peerLogger.Warningf("Failed to find config block on ledger %s(%s)", cid, err)
peerLogger.Debugf("Error while looking for config block on ledger %s with message %s. We continue to the next ledger rather than abort.", cid, err)
continue
}
// Create a chain if we get a valid ledger with config block
if err = createChain(cid, ledger, cb); err != nil {
peerLogger.Warningf("Failed to load chain %s(%s)", cid, err)
peerLogger.Debugf("Error reloading chain %s with message %s. We continue to the next chain rather than abort.", cid, err)
continue
}
InitChain(cid)
}
}
```
dave.enyeart (Tue, 07 Mar 2017 12:51:02 GMT):
ok. and if there is a later config update, you get this info from the later config block, or always from genesis block?
C0rWin (Tue, 07 Mar 2017 12:51:06 GMT):
this is the code of peer initialization right?
C0rWin (Tue, 07 Mar 2017 12:51:53 GMT):
so it basically iterates over ledgers and recreates the infrastructure entities to allow peer to operate across all channels it was subscribed to
C0rWin (Tue, 07 Mar 2017 12:52:01 GMT):
so if you won't have GB commited
C0rWin (Tue, 07 Mar 2017 12:52:17 GMT):
restarting the peer you will have to somehow resubmit the GB
C0rWin (Tue, 07 Mar 2017 12:55:12 GMT):
if there is later config block you will read update from it
dave.enyeart (Tue, 07 Mar 2017 12:55:18 GMT):
makes sense.... with current flow it is possible for ledger to get created, but no genesis block, since they are separate calls:
dave.enyeart (Tue, 07 Mar 2017 12:55:23 GMT):
```// CreateChainFromBlock creates a new chain from config block
func CreateChainFromBlock(cb *common.Block) error {
cid, err := utils.GetChainIDFromBlock(cb)
if err != nil {
return err
}
var ledger ledger.PeerLedger
if ledger, err = createLedger(cid); err != nil {
return err
}
if err := ledger.Commit(cb); err != nil {
peerLogger.Errorf("Unable to get genesis block committed into the ledger, chainID %v", cid)
return err
}
return createChain(cid, ledger, cb)
}```
C0rWin (Tue, 07 Mar 2017 12:55:40 GMT):
ok
C0rWin (Tue, 07 Mar 2017 12:56:18 GMT):
but, probably the intermediate solution will be to destroy ledger if you cannot commit the block
dave.enyeart (Tue, 07 Mar 2017 12:56:25 GMT):
probably createLedger needs to accept genesis block, so that we can be in a consistent state (all success or all fail)
C0rWin (Tue, 07 Mar 2017 12:56:39 GMT):
since anyhow you will restart the peer to solve the problem, right?
C0rWin (Tue, 07 Mar 2017 12:56:59 GMT):
there is an error state
dave.enyeart (Tue, 07 Mar 2017 12:57:38 GMT):
right, its just a question of whether the 'rollback' should be handled in this joinChain code, or abstracted within the createLedger() code
C0rWin (Tue, 07 Mar 2017 12:57:56 GMT):
ok
C0rWin (Tue, 07 Mar 2017 12:58:45 GMT):
so basically you trying to achieve the ability in case of problem with ledger, to restart the peer and get the GB via gossip w/o a need to explicitly call join channel once again?
dave.enyeart (Tue, 07 Mar 2017 12:59:15 GMT):
just exploring if that is an option
C0rWin (Tue, 07 Mar 2017 12:59:31 GMT):
I'd say we have to be very careful here, since in that case I'm not sure peer will be able to build gossip membership
C0rWin (Tue, 07 Mar 2017 12:59:48 GMT):
membership build based on the anchor peers info
manish-sethi (Tue, 07 Mar 2017 12:59:53 GMT):
@C0rWin: In addition to `Initialize` that initializes the existing chain, does gossip also take care of the chains that are supposed to be on a peer (e.g., peer was down when the chain got created)?
C0rWin (Tue, 07 Mar 2017 12:59:54 GMT):
so if no GB available
dave.enyeart (Tue, 07 Mar 2017 13:00:27 GMT):
you have to explicitly join() each peer when the peer is up
C0rWin (Tue, 07 Mar 2017 13:00:32 GMT):
> In addition to `Initialize` that initializes the existing chain, does gossip also take care of the chains that are supposed to be on a peer (e.g., peer was down when the chain got created)?
How that even possible?
C0rWin (Tue, 07 Mar 2017 13:00:38 GMT):
peer was down...
C0rWin (Tue, 07 Mar 2017 13:00:49 GMT):
there is no magic in here
C0rWin (Tue, 07 Mar 2017 13:00:54 GMT):
peer is running gossip code
dave.enyeart (Tue, 07 Mar 2017 13:01:09 GMT):
i think Manish did not know that an explicit join() was required
manish-sethi (Tue, 07 Mar 2017 13:01:11 GMT):
so, who keeps track what all peers are supposed to be part of a chain?
dave.enyeart (Tue, 07 Mar 2017 13:02:04 GMT):
orderering service has the membership info in genesis block
C0rWin (Tue, 07 Mar 2017 13:02:25 GMT):
no-no-no
manish-sethi (Tue, 07 Mar 2017 13:02:30 GMT):
and peer explicitly raise a join request by some manual operation?
C0rWin (Tue, 07 Mar 2017 13:02:35 GMT):
ordering service has membership on the org level
C0rWin (Tue, 07 Mar 2017 13:02:49 GMT):
no one actually has the global knowledge which peer has to be part of the channel
dave.enyeart (Tue, 07 Mar 2017 13:03:11 GMT):
every peer is part of an org, which is part of a membership, right
C0rWin (Tue, 07 Mar 2017 13:03:13 GMT):
> and peer explicitly raise a join request by some manual operation?
yes, you need explicitly call join channel for peer
C0rWin (Tue, 07 Mar 2017 13:03:43 GMT):
> every peer is part of an org, which is part of a membership, right
correct, but you cannot tell which peers included in the channel
C0rWin (Tue, 07 Mar 2017 13:03:55 GMT):
based on the info from ordering service
manish-sethi (Tue, 07 Mar 2017 13:04:06 GMT):
OK, and during that explicit join call, you obtain a GB and create a new ledger
dave.enyeart (Tue, 07 Mar 2017 13:04:13 GMT):
you mean, which peers 'should' be part of the channel? or which peers have actually joined the channel?
C0rWin (Tue, 07 Mar 2017 13:04:26 GMT):
both
dave.enyeart (Tue, 07 Mar 2017 13:04:46 GMT):
can't you tell from the peer's org which channel's it is authorized to join?
C0rWin (Tue, 07 Mar 2017 13:05:42 GMT):
yes
C0rWin (Tue, 07 Mar 2017 13:05:58 GMT):
but that doesn't mean this peer has to be in that channel
C0rWin (Tue, 07 Mar 2017 13:06:10 GMT):
in that sense if org A part of the channel XXX
C0rWin (Tue, 07 Mar 2017 13:06:17 GMT):
every single peer from org A
dave.enyeart (Tue, 07 Mar 2017 13:06:24 GMT):
ok, peer has the option to join the channel
manish-sethi (Tue, 07 Mar 2017 13:06:37 GMT):
So, join is a manual operation initiated at a peer and during that explicit join call, you obtain a GB and create a new ledger?
C0rWin (Tue, 07 Mar 2017 13:06:37 GMT):
allowed to be part of that channel
dave.enyeart (Tue, 07 Mar 2017 13:07:23 GMT):
correct manish, so i think the proper design would be for createLedger() to accept the genesis block and it either atomically succeeds or fails
manish-sethi (Tue, 07 Mar 2017 13:08:07 GMT):
Yes, that's right.
dave.enyeart (Tue, 07 Mar 2017 13:08:45 GMT):
ok, we can go work that offline and free up Artem :)
manish-sethi (Tue, 07 Mar 2017 13:09:22 GMT):
Yes, sure
dave.enyeart (Tue, 07 Mar 2017 13:09:40 GMT):
I suspect this would be a post-beta enhancement
C0rWin (Tue, 07 Mar 2017 13:12:07 GMT):
@dave.enyeart @manish-sethi I'd be happy to follow up on this later
manish-sethi (Tue, 07 Mar 2017 13:12:33 GMT):
From ledger changes point of view, it should take a couple of days. The changes in the higher layer should be easy because even now mostly it would have been create ledger and commit GB in continuous statements
dave.enyeart (Tue, 07 Mar 2017 13:13:04 GMT):
correct. currently it looks like this:
dave.enyeart (Tue, 07 Mar 2017 13:13:09 GMT):
```// CreateChainFromBlock creates a new chain from config block
func CreateChainFromBlock(cb *common.Block) error {
cid, err := utils.GetChainIDFromBlock(cb)
if err != nil {
return err
}
var ledger ledger.PeerLedger
if ledger, err = createLedger(cid); err != nil {
return err
}
if err := ledger.Commit(cb); err != nil {
peerLogger.Errorf("Unable to get genesis block committed into the ledger, chainID %v", cid)
return err
}
return createChain(cid, ledger, cb)
}```
dave.enyeart (Tue, 07 Mar 2017 13:13:46 GMT):
we need to simply combine createLedger and commit of genesis block into one atomic function call
manish-sethi (Tue, 07 Mar 2017 13:14:34 GMT):
yes
dave.enyeart (Tue, 07 Mar 2017 13:14:53 GMT):
we should not introduce this churn for beta... therefore we have time and can do it post-beta
muralisr (Tue, 07 Mar 2017 13:15:23 GMT):
thanks @dave.enyeart :-)
manish-sethi (Tue, 07 Mar 2017 13:15:59 GMT):
sure, Murali's joined hands are making my lol :-)
dave.enyeart (Tue, 07 Mar 2017 13:16:26 GMT):
Murali has enough problems on his hands without us introducing churn :)
manish-sethi (Tue, 07 Mar 2017 13:16:40 GMT):
yes, indeed
C0rWin (Tue, 07 Mar 2017 13:16:50 GMT):
> we need to simply combine createLedger and commit of genesis block into one atomic function call
```
err := createLedger(gb)
```
^^^ I would imagine it will look like this?
dave.enyeart (Tue, 07 Mar 2017 13:16:57 GMT):
not problems of his own making :)
dave.enyeart (Tue, 07 Mar 2017 13:17:08 GMT):
yes
dave.enyeart (Tue, 07 Mar 2017 13:17:41 GMT):
createLedger(cid, gb)
C0rWin (Tue, 07 Mar 2017 13:17:52 GMT):
cid part of gb
C0rWin (Tue, 07 Mar 2017 13:17:57 GMT):
looks redundant
C0rWin (Tue, 07 Mar 2017 13:17:58 GMT):
no?
dave.enyeart (Tue, 07 Mar 2017 13:18:14 GMT):
you dont want to force ledger to parse config :)
C0rWin (Tue, 07 Mar 2017 13:18:29 GMT):
ok
dave.enyeart (Tue, 07 Mar 2017 13:18:41 GMT):
ledger is 'dumb' wrt config
manish-sethi (Tue, 07 Mar 2017 13:18:54 GMT):
yes, but before calling this, the caller would have to check 'exist(ledgerID)' because, a crash may have happened during a join operation but join actually succeeded at the ledger level.
C0rWin (Tue, 07 Mar 2017 13:19:23 GMT):
@dave.enyeart is there JIRA item for this effort?
dave.enyeart (Tue, 07 Mar 2017 13:19:32 GMT):
i will create it now
manish-sethi (Tue, 07 Mar 2017 13:19:58 GMT):
So, at next join request, it should be checked first whether the last time the ledger create was succeeded
C0rWin (Tue, 07 Mar 2017 13:20:33 GMT):
I just want to subscribe to it to follow the development of this
manish-sethi (Tue, 07 Mar 2017 13:21:40 GMT):
@C0rWin just curious what other components (in addition to ledger) are involved in join operation. They also maintain some state or for them it doesn't matter?
C0rWin (Tue, 07 Mar 2017 13:22:47 GMT):
@manish-sethi MSP configs loaded from GB
C0rWin (Tue, 07 Mar 2017 13:23:23 GMT):
I'd guess it also involved in join operation in some way or another
dave.enyeart (Tue, 07 Mar 2017 13:24:49 GMT):
https://jira.hyperledger.org/browse/FAB-2676 : Need to atomically createLedger and commit genesis block
manish-sethi (Tue, 07 Mar 2017 13:25:09 GMT):
So, from ledger point of view the two calls `Exists(ledgerId)` (which is already there) and `Create(GB)` should be enough? Even if a crash happens during a join and a join is re-initiated after a restart?
dave.enyeart (Tue, 07 Mar 2017 13:26:34 GMT):
i believe that's right
manish-sethi (Tue, 07 Mar 2017 13:27:32 GMT):
@C0rWin - does that sound reasonable to you? I am asking because you have better visibility about the overall join operation
yacovm (Tue, 07 Mar 2017 13:29:30 GMT):
Just chiming in to say that it is design to be impossible to gossip a genesis block of a channel
yacovm (Tue, 07 Mar 2017 13:29:30 GMT):
Just chiming in to say that it is designed to be impossible to gossip a genesis block of a channel
dave.enyeart (Tue, 07 Mar 2017 13:29:57 GMT):
ok, further confirms our decision :)
manish-sethi (Tue, 07 Mar 2017 13:32:24 GMT):
OK, @yacovm. whenever you get time... It would be good if you or Artem can confirm that the above two APIs are good enough for the join operation to survive a crash.
yacovm (Tue, 07 Mar 2017 13:40:20 GMT):
I think Artem is right
yacovm (Tue, 07 Mar 2017 13:40:30 GMT):
Message Attachments
yacovm (Tue, 07 Mar 2017 13:41:00 GMT):
if the commit fails,the peer shouldn't advance further in the join-channel flow
yacovm (Tue, 07 Mar 2017 13:43:35 GMT):
My personal opinion is that if the commit fails it's probably because the DB isn't accessible, right?
So the join operation should be synchronous and block until it succeeds, and it should return a failure status all from that `joinChain` operation all the way back to the gRPC connection of the CLI/SDK with a failure status
yacovm (Tue, 07 Mar 2017 13:43:35 GMT):
My personal opinion is that the join operation should be synchronous and block until it succeeds, and it should return a failure status all from that `joinChain` operation all the way back to the gRPC connection of the CLI/SDK with a failure status
yacovm (Tue, 07 Mar 2017 13:43:35 GMT):
My personal opinion is that the join operation should be synchronous and block until it succeeds, and it should return a failure status from that `joinChain` operation all the way back to the gRPC connection of the CLI/SDK with a failure status
C0rWin (Tue, 07 Mar 2017 13:48:53 GMT):
> So, from ledger point of view the two calls `Exists(ledgerId)` (which is already there) and `Create(GB)` should be enough? Even if a crash happens during a join and a join is re-initiated after a restart?
^^^ Can you please elaborate how do you envision the restart flow?
C0rWin (Tue, 07 Mar 2017 13:49:20 GMT):
What exactly do you assume? @dave.enyeart @manish-sethi ^^^^
dave.enyeart (Tue, 07 Mar 2017 13:58:52 GMT):
I think we'll have to get further into the detailed design and then get back to you on recovery scenarios
C0rWin (Tue, 07 Mar 2017 14:15:13 GMT):
sounds, good, so once you will figure out details will be able to confirm whenever proposed two API's are sufficient to handle crash recovery
manish-sethi (Tue, 07 Mar 2017 16:07:54 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=QJYQQMmTu8Dd5M33k) @C0rWin What I assume is that when a join operation is initiated, first call should be `Exists(ledgerId)` - which if returns true that means the ledger is already created - most likely in a previous attempt during which peer may have crashed. If the `Exists` returns false then `Create(GB)` call should be made. If this call return successfully then ledger is created otherwise not.
xixuejia (Wed, 08 Mar 2017 14:47:12 GMT):
Hi @yacovm a few questions about gossip 1)is the gossip leader the only peer that calls ordering service Deliver in an organization? 2) will Gossip layer take endorser the same as committer or differently when selecting the gossip leader inside an organization?
yacovm (Wed, 08 Mar 2017 14:53:24 GMT):
1) Yes
2) There is nothing in gossip that relates to endorsements
xixuejia (Wed, 08 Mar 2017 15:03:24 GMT):
thanks!
xixuejia (Wed, 08 Mar 2017 15:09:04 GMT):
I asked question 2 because I think endorser should always keep latest state inside an org, (being gossip leader would help), though this is not critical
Suma (Wed, 08 Mar 2017 15:43:10 GMT):
Has joined the channel.
berserkr (Thu, 09 Mar 2017 00:54:42 GMT):
Has joined the channel.
dave.enyeart (Thu, 09 Mar 2017 17:04:38 GMT):
@yacovm @C0rWin I believe committing peer can process blocks from various channels in parallel, and call ledger.Commit() in parallel (one thread with one Commit() ongoing per channel). @Senthil1 was asking how this worked. And I as well would like to understand the high level gossip procedure for block dissemination from orderer to committing peer (we can assume all peers are leaders for now, to keep the answer simple).
Senthil1 (Thu, 09 Mar 2017 17:04:38 GMT):
Has joined the channel.
dave.enyeart (Thu, 09 Mar 2017 17:15:45 GMT):
If this is documented somewhere, feel free to point me :)
C0rWin (Thu, 09 Mar 2017 17:58:30 GMT):
@dave.enyeart not sure what exactly you are looking for, while dissemination procedure is very simple. Gossip leaders pulls out blocks from ordering service and distribute them across other peers. If peers receives same block which was already processed twice it simply drops it.
dave.enyeart (Thu, 09 Mar 2017 18:03:21 GMT):
@C0rWin I'm looking for the multi-threading behavior. Can the blocks be pulled from ordering service and committed to ledger in parallel? I'm assuming one thread doing this in parallel per channel, right?
C0rWin (Thu, 09 Mar 2017 18:26:15 GMT):
@dave.enyeart pulling of blocks out of ordering service and block dissemination two decoupled processes same for block commits
dave.enyeart (Thu, 09 Mar 2017 18:26:59 GMT):
ok, this is what we want to understand better. what is the sequence of processes, and which processes can be done in parallel?
C0rWin (Thu, 09 Mar 2017 18:29:47 GMT):
@dave.enyeart can you please be more specific? sending blocks, processing blocks to the ledger and pulling blocks out of ordering service all this are parallel processes
dave.enyeart (Thu, 09 Mar 2017 18:32:23 GMT):
I mean, what is the degree of parallelization? I assume you get blocks for ordering service one at a time, and then commit it to ledger, and then get next block from orderering service? And this process is done in parallel for each channel?
dave.enyeart (Thu, 09 Mar 2017 18:32:58 GMT):
Looking for the one paragraph high level explanation of the flow.
berserkr (Thu, 09 Mar 2017 18:36:58 GMT):
if there are 3 peer nodes, they all get the blocks in parallel right?
berserkr (Thu, 09 Mar 2017 18:37:11 GMT):
peer nodes would do things sequentially per channel
berserkr (Thu, 09 Mar 2017 18:38:03 GMT):
parallelization happens at the peer level rather than thread level right?
dave.enyeart (Thu, 09 Mar 2017 18:39:52 GMT):
yes, 3 leader peers would process blocks in parallel. Let's understand from @C0rWin the sequencing/parallelization within each leader peer.
Senthil1 (Thu, 09 Mar 2017 18:47:41 GMT):
I went through the code to understand the steps: What i understood is the following (correct me if I am wrong).
(1) `InitializeChannel()` in _gossip/service/gossip_service.go_ is called to create a channel which in turn calls `NewGossipStateProvider()` in _gossip/state/state.go_ and `StartDeliveryForChannel()` in _core/deliverservice/deliveryclient.go_
(2) As a result, 4 go routines (i.e., *threads*) are created for this channel: `listen()`, `deliverPayloads()`, `antiEntropy()`, `DeliverBlocks()`
(3) `DeliverBlocks()` in _core/deliverservice/blocksprovider/blocksprovider.go_ fetches block from ordering service for that channel and gossip the block to other peers (non-leaders)
(4) `deliverPayloads()` in _gossip/state/state.go_ , on receiving a block, calls committer to commit the block.
What we wanted to know was the point (2).
yacovm (Thu, 09 Mar 2017 18:54:47 GMT):
What do you mean "was the point"?
berserkr (Thu, 09 Mar 2017 18:55:43 GMT):
4 threads are created, yes, but each does something different
berserkr (Thu, 09 Mar 2017 18:55:54 GMT):
not what we are asking
dave.enyeart (Thu, 09 Mar 2017 19:02:58 GMT):
@yacovm The question remains - what is the degree of parallelization (for deliverBlocks() and deliverPayloads()? I assume you get blocks for ordering service one at a time, and then commit it to ledger, and then get next block from orderering service? And this process is done in parallel for each channel?
yacovm (Thu, 09 Mar 2017 19:03:24 GMT):
Willl answer once i get home
dave.enyeart (Thu, 09 Mar 2017 19:03:29 GMT):
no hurry!
C0rWin (Thu, 09 Mar 2017 19:04:59 GMT):
@dave.enyeart process of pulling blocks and committing to the ledger completely decoupled, the communication between these two via queueing and yes this done in parallel for each channel
C0rWin (Thu, 09 Mar 2017 19:05:40 GMT):
are asking because this because you trying to see whenever you need to take care of synchronization and locks while blocks are committed?
C0rWin (Thu, 09 Mar 2017 19:05:40 GMT):
are you asking because this because you trying to see whenever you need to take care of synchronization and locks while blocks are committed?
C0rWin (Thu, 09 Mar 2017 19:05:40 GMT):
are you asking because you trying to see whenever you need to take care of synchronization and locks while blocks are committed?
berserkr (Thu, 09 Mar 2017 19:07:18 GMT):
each peer has a queue, one thread pulls block, puts on queue, anoher thread reads block from queue, commits to ledger
berserkr (Thu, 09 Mar 2017 19:07:26 GMT):
that is the level of parallelism right?
C0rWin (Thu, 09 Mar 2017 19:08:01 GMT):
yes
berserkr (Thu, 09 Mar 2017 19:08:10 GMT):
not: channel 1 has 2 threads, channel 2 has two threads, channel 3 has 2 threads...
berserkr (Thu, 09 Mar 2017 19:08:12 GMT):
ok
berserkr (Thu, 09 Mar 2017 19:08:14 GMT):
makes sense
dave.enyeart (Thu, 09 Mar 2017 19:08:38 GMT):
Thanks @C0rWin, that answers the question. We are doing some perf test and want to understand the parallelism.
dave.enyeart (Thu, 09 Mar 2017 19:08:38 GMT):
Thanks @C0rWin, that answers the question. We are doing some perf test and wanted to understand the parallelism.
dave.enyeart (Thu, 09 Mar 2017 19:08:38 GMT):
Thanks @C0rWin, that answers the question. We are doing some perf test and wanted to understand the parallelism. Ensure we are getting the perf benefits of multi-channel, etc.
Senthil1 (Thu, 09 Mar 2017 19:13:07 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=pCiQbrscXyrDxN8Ma) @yacovm
(2) As a result, 4 go routines (i.e., *threads*) are created ~this~ per channel `listen()`, `deliverPayloads()`, `antiEntropy()`, `DeliverBlocks()`
I meant that I was looking for the above answer.
Senthil1 (Thu, 09 Mar 2017 19:13:07 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=pCiQbrscXyrDxN8Ma) @yacovm
(2) As a result, 4 go routines (i.e., *threads*) are created ~this~ per channel `listen()`, `deliverPayloads()`, `antiEntropy()`, `DeliverBlocks()`
I meant that I was looking for the above answer.
Senthil1 (Thu, 09 Mar 2017 19:13:07 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=pCiQbrscXyrDxN8Ma) @yacovm
(2) As a result, 4 go routines (i.e., *threads*) are created ~this~ per channel `listen()`, `deliverPayloads()`, `antiEntropy()`, `DeliverBlocks()`
I meant that I was looking for the above answer (to understand the parallelism) .
Senthil1 (Thu, 09 Mar 2017 19:14:12 GMT):
Thanks @C0rWin for the clarification.
berserkr (Thu, 09 Mar 2017 19:18:26 GMT):
@Senthil1 so it is the latter, per channel? so you have 4 channels, a total of 4 x4 threads are created?
Senthil1 (Thu, 09 Mar 2017 19:19:46 GMT):
Yes.
berserkr (Thu, 09 Mar 2017 19:21:25 GMT):
interesting, thanks for the clarification @Senthil1 @C0rWin
samirsadeghi (Thu, 09 Mar 2017 20:02:28 GMT):
Has joined the channel.
scottz (Thu, 09 Mar 2017 20:51:48 GMT):
Has joined the channel.
scottz (Thu, 09 Mar 2017 20:58:04 GMT):
In v1.0 today, should things work if I bring up a network of peers where all of them set CORE_PEER_GOSSIP_ORGLEADER=false. Will they self elect? Or do I misunderstand how this is supposed to work?
yacovm (Thu, 09 Mar 2017 20:59:01 GMT):
@scottz you can turn on leader election
yacovm (Thu, 09 Mar 2017 20:59:03 GMT):
and then they will
yacovm (Thu, 09 Mar 2017 20:59:59 GMT):
https://github.com/hyperledger/fabric/blob/master/peer/core.yaml#L74-L77
scottz (Thu, 09 Mar 2017 21:01:20 GMT):
cool. for every peer, right?
yacovm (Thu, 09 Mar 2017 21:03:57 GMT):
yes
scottz (Thu, 09 Mar 2017 21:07:36 GMT):
Thanks. Trying it. And Wow! There are lots of config parameters...
yacovm (Thu, 09 Mar 2017 21:08:12 GMT):
well
yacovm (Thu, 09 Mar 2017 21:08:21 GMT):
please don't press the red button
yacovm (Thu, 09 Mar 2017 21:08:26 GMT):
whatever you do ;)
yacovm (Thu, 09 Mar 2017 21:10:53 GMT):
BTW- all the "time-durations" need to be equal across all peers
C0rWin (Thu, 09 Mar 2017 21:15:17 GMT):
I'd say that most of the parameters there is to tune gossip performance
C0rWin (Thu, 09 Mar 2017 21:15:58 GMT):
so they configured to the proper values and this is very important to keep they consistent across peers
scottz (Thu, 09 Mar 2017 22:25:02 GMT):
thanks. and by the way it worked with self election. but you probably knew that would work...
JatinderBali (Fri, 10 Mar 2017 15:44:08 GMT):
Has joined the channel.
AdnanC (Fri, 10 Mar 2017 20:30:29 GMT):
Hi, @rahulhegde was wondering how to reduce the logs from gossip component in the peer log.
rahulhegde (Fri, 10 Mar 2017 20:37:50 GMT):
True - I would like to know how can reduce logs of gossip at peer startup. I tried using peer logging setlevel
rahulhegde (Fri, 10 Mar 2017 20:37:50 GMT):
True - I would like to know how can reduce logs level (to ERROR) of gossip at peer startup. I tried using peer logging setlevel
yacovm (Fri, 10 Mar 2017 21:14:00 GMT):
what do you mean goes in a hung state? the command doesn't end?
rahulhegde (Fri, 10 Mar 2017 21:20:22 GMT):
yes @yacom
yacovm (Fri, 10 Mar 2017 21:22:50 GMT):
that's a bug, perhaps @wlahti knows or can help troubleshoot as he implemented this
wlahti (Fri, 10 Mar 2017 21:22:50 GMT):
Has joined the channel.
wlahti (Fri, 10 Mar 2017 21:26:36 GMT):
hmmm, very odd. I've never seen that or heard of any issues. what kind of setup are you using, @rahulhegde? docker containers? I don't use the setlevel command all that often these days but it was definitely working for me late last week.
rahulhegde (Fri, 10 Mar 2017 21:35:28 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=JiPi6nTnFFXTTjNRj) @wlahti
This is Fabric CI Channel provided 4th March Images which has scrolling GOSSIP related logs.
Dave gave me -
peer logging setlevel gossip/comm#-1 ERROR
peer logging setlevel gossip/discovery#0.0.0.0:7051 ERROR
peer logging setlevel gossip/gossip#0.0.0.0:7051 ERROR
rahulhegde (Fri, 10 Mar 2017 21:35:28 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=JiPi6nTnFFXTTjNRj) @wlahti
This is Fabric CI Channel provided 4th March Images which has scrolling GOSSIP related logs.
Dave gave me -
peer logging setlevel gossip/comm#-1 ERROR
peer logging setlevel gossip/discovery#0.0.0.0:7051 ERROR
peer logging setlevel gossip/gossip#0.0.0.0:7051 ERROR
` peer logging setlevel gossip/discovery#0.0.0.0:7051 ERROR ` went in a hang state.
rahulhegde (Fri, 10 Mar 2017 21:35:28 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=JiPi6nTnFFXTTjNRj) @wlahti
This is Fabric CI Channel provided 4th March Images which has scrolling GOSSIP related logs.
Dave gave me -
peer logging setlevel gossip/comm#-1 ERROR
peer logging setlevel gossip/discovery#0.0.0.0:7051 ERROR
peer logging setlevel gossip/gossip#0.0.0.0:7051 ERROR
` peer logging setlevel gossip/comm#-1 ERROR ` went through well - but I could see logs scrolling from the logger.
` peer logging setlevel gossip/discovery#0.0.0.0:7051 ERROR ` went in a hang state.
rahulhegde (Fri, 10 Mar 2017 21:35:28 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=JiPi6nTnFFXTTjNRj) @wlahti
This is Fabric CI Channel provided 4th March Images which has scrolling GOSSIP related logs.
Dave gave me -
peer logging setlevel gossip/comm#-1 ERROR
peer logging setlevel gossip/discovery#0.0.0.0:7051 ERROR
peer logging setlevel gossip/gossip#0.0.0.0:7051 ERROR
` peer logging setlevel gossip/comm#-1 ERROR ` went through well - but I could see logs scrolling from this instance logger.
` peer logging setlevel gossip/discovery#0.0.0.0:7051 ERROR ` went in a hang state.
yacovm (Fri, 10 Mar 2017 21:36:38 GMT):
Will how is your configurable-per-subcomponent JIRA item going?
yacovm (Fri, 10 Mar 2017 21:36:48 GMT):
I recall Dave/Binh assigned it to you no?
yacovm (Fri, 10 Mar 2017 21:36:56 GMT):
@wlahti
yacovm (Fri, 10 Mar 2017 21:40:31 GMT):
It can solve rahulhedge's problem
yacovm (Fri, 10 Mar 2017 21:40:31 GMT):
It can solve rahulhegde's problem
wlahti (Fri, 10 Mar 2017 21:41:53 GMT):
I haven't had much time to look into the subcomponent JIRA yet. Other things have taken priority.
rickr (Mon, 13 Mar 2017 02:18:17 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=2kBYpst3BqpcjidkD) Here has been my goto solution: docker-compose up --force-recreate * 2>&1 | grep -v 'UTC \[gossip/'*
antoniovassell (Mon, 13 Mar 2017 16:30:41 GMT):
Has joined the channel.
MadhavaReddy (Tue, 14 Mar 2017 03:02:48 GMT):
Has joined the channel.
MadhavaReddy (Tue, 14 Mar 2017 03:04:49 GMT):
Hi All, i started 1.0 using docker-compose file which is present /e2c_cli, though all containers are running however peers are not doing anything, the only log that i see in peer0 is
MadhavaReddy (Tue, 14 Mar 2017 03:04:51 GMT):
2017-03-14 02:25:08.945 UTC [gossip/comm#-1] GossipStream -> ERRO 00f Authentication failed
2017-03-14 02:25:09.227 UTC [gossip/comm#-1] GossipStream -> ERRO 010 Authentication failed
2017-03-14 02:25:33.948 UTC [gossip/comm#-1] GossipStream -> ERRO 011 Authentication failed
2017-03-14 02:25:34.230 UTC [gossip/comm#-1] GossipStream -> ERRO 012 Authentication failed
2017-03-14 02:25:58.950 UTC [gossip/comm#-1] GossipStream -> ERRO 013 Authentication failed
2017-03-14 02:25:59.232 UTC [gossip/comm#-1] GossipStream -> ERRO 014 Authentication failed
2017-03-14 02:26:23.969 UTC [gossip/comm#-1] GossipStream -> ERRO 015 Authentication failed
2017-03-14 02:26:24.235 UTC [gossip/comm#-1] GossipStream -> ERRO 016 Authentication failed
MadhavaReddy (Tue, 14 Mar 2017 03:05:32 GMT):
above, apart from this no entries found in other peer container logs, could you please help me to fix this issue
yacovm (Tue, 14 Mar 2017 08:01:55 GMT):
What do you see in logs of peers 1-3?
ruslan.kryukov (Tue, 14 Mar 2017 15:54:40 GMT):
Has joined the channel.
a-p-g (Tue, 14 Mar 2017 16:43:22 GMT):
Has joined the channel.
lignyxg (Tue, 14 Mar 2017 17:32:39 GMT):
Has joined the channel.
MadhavaReddy (Tue, 14 Mar 2017 19:47:30 GMT):
@yacovm not even single entry in other peer logs
MadhavaReddy (Tue, 14 Mar 2017 19:48:04 GMT):
All i again tried to "docker-compose-no-tls.yaml" but still the same issue
MadhavaReddy (Tue, 14 Mar 2017 19:48:09 GMT):
2017-03-14 19:44:06.951 UTC [gossip/comm#-1] GossipStream -> ERRO 001 Authentication failed
2017-03-14 19:44:07.267 UTC [gossip/comm#-1] GossipStream -> ERRO 002 Authentication failed
2017-03-14 19:44:31.953 UTC [gossip/comm#-1] GossipStream -> ERRO 003 Authentication failed
2017-03-14 19:44:32.271 UTC [gossip/comm#-1] GossipStream -> ERRO 004 Authentication failed
2017-03-14 19:44:56.956 UTC [gossip/comm#-1] GossipStream -> ERRO 005 Authentication failed
2017-03-14 19:44:57.273 UTC [gossip/comm#-1] GossipStream -> ERRO 006 Authentication failed
2017-03-14 19:45:21.973 UTC [gossip/comm#-1] GossipStream -> ERRO 007 Authentication failed
2017-03-14 19:45:22.276 UTC [gossip/comm#-1] GossipStream -> ERRO 008 Authentication failed
2017-03-14 19:45:46.976 UTC [gossip/comm#-1] GossipStream -> ERRO 009 Authentication failed
2017-03-14 19:45:47.278 UTC [gossip/comm#-1] GossipStream -> ERRO 00a Authentication failed
2017-03-14 19:46:11.978 UTC [gossip/comm#-1] GossipStream -> ERRO 00b Authentication failed
2017-03-14 19:46:12.281 UTC [gossip/comm#-1] GossipStream -> ERRO 00c Authentication failed
2017-03-14 19:46:36.999 UTC [gossip/comm#-1] GossipStream -> ERRO 00d Authentication failed
2017-03-14 19:46:37.285 UTC [gossip/comm#-1] GossipStream -> ERRO 00e Authentication failed
MadhavaReddy (Tue, 14 Mar 2017 19:48:43 GMT):
i only see auth failed and not even a single entry in other peer containers
MadhavaReddy (Tue, 14 Mar 2017 19:49:01 GMT):
please help me resolve the issue
yacovm (Tue, 14 Mar 2017 19:54:47 GMT):
can you turn on debug in all peers and try again?
ruslan.kryukov (Wed, 15 Mar 2017 08:51:59 GMT):
If I start fabric with genesis method file, gossip spams to log with message:
peer-01 | 2017-03-15 08:49:54.127 UTC [gossip/gossip#IP:7051] PeersOfChannel -> WARN 094 No such channel [116 101 115 116 99 104 97 105 110 105 100]
116 101 bla bla bla - is testchainid word
ruslan.kryukov (Wed, 15 Mar 2017 08:55:14 GMT):
my config block contains my own channel name and orderer in this case doesn't know about testchainid
But when I start peer with command "peer node start", it starts with default channel testchainid, right? After that I send command "peer join" to my channel, but gossip still sends a lot of messages "no such channel"
yacovm (Wed, 15 Mar 2017 08:56:41 GMT):
yeah
yacovm (Wed, 15 Mar 2017 08:56:54 GMT):
because if you start without the --testchainid false flag
yacovm (Wed, 15 Mar 2017 08:57:03 GMT):
the peers announce that they are part of "testchainid"
yacovm (Wed, 15 Mar 2017 08:57:25 GMT):
` --peer-defaultchain=false`
yacovm (Wed, 15 Mar 2017 08:57:30 GMT):
use this when starting the peer
yacovm (Wed, 15 Mar 2017 09:07:07 GMT):
@ruslan.kryukov also you can `git pull` now and rebuild the peer, you won't see these messages anymore.
Thanks @mastersingh24 for the merge
ruslan.kryukov (Wed, 15 Mar 2017 09:08:06 GMT):
Oh, good, thanks
mychewcents (Wed, 15 Mar 2017 12:28:56 GMT):
Has joined the channel.
RahulBagaria (Fri, 17 Mar 2017 04:02:27 GMT):
Has joined the channel.
MadhavaReddy (Fri, 17 Mar 2017 16:48:10 GMT):
Hi All am trying to run 1.0 env from e2e_cli folder using no tls yaml file and am seeing below error in peer docker logs, i ran generateCfgTrx.sh script before i ran docker-compose cmd, please note i comment script in yaml file
MadhavaReddy (Fri, 17 Mar 2017 16:48:22 GMT):
2017-03-17 16:44:47.681 UTC [gossip/comm#-1] GossipStream -> ERRO 196 Authentication failed
2017-03-17 16:44:50.557 UTC [gossip/pull#172.19.0.3:7051] Hello -> DEBU 197 Sending hello to 172.19.0.4:7051
2017-03-17 16:44:50.557 UTC [gossip/comm#-1] Send -> DEBU 198 Entering, sending GossipMessage: tag:EMPTY hello:
MadhavaReddy (Fri, 17 Mar 2017 16:48:53 GMT):
please help me on this issue
yacovm (Fri, 17 Mar 2017 16:53:46 GMT):
What's the issue?
MadhavaReddy (Fri, 17 Mar 2017 16:54:45 GMT):
2017-03-17 16:44:47.681 UTC [gossip/comm#-1] GossipStream -> ERRO 196 Authentication failed
MadhavaReddy (Fri, 17 Mar 2017 16:55:19 GMT):
and when i try to create new channel in CLI container
MadhavaReddy (Fri, 17 Mar 2017 16:55:26 GMT):
root@72a84d8de556:/opt/gopath/src/github.com/hyperledger/fabric/peer# peer channel create -o orderer0:7050 -c mychannel -f crypto/orderer/channel.tx
2017-03-17 16:49:51.090 UTC [logging] InitFromViper -> DEBU 001 Setting default logging level to DEBUG for command 'channel'
2017-03-17 16:49:51.090 UTC [msp] GetLocalMSP -> DEBU 002 Returning existing local MSP
2017-03-17 16:49:51.090 UTC [msp] GetDefaultSigningIdentity -> DEBU 003 Obtaining default signing identity
2017-03-17 16:49:53.127 UTC [msp] GetLocalMSP -> DEBU 004 Returning existing local MSP
2017-03-17 16:49:53.127 UTC [msp] GetDefaultSigningIdentity -> DEBU 005 Obtaining default signing identity
2017-03-17 16:49:53.127 UTC [msp] GetLocalMSP -> DEBU 006 Returning existing local MSP
2017-03-17 16:49:53.127 UTC [msp] GetDefaultSigningIdentity -> DEBU 007 Obtaining default signing identity
2017-03-17 16:49:53.127 UTC [msp] Sign -> DEBU 008 Sign: plaintext: 0ADD070A1508021A0608B1A7B0C60522...00120D1A0B08FFFFFFFFFFFFFFFFFF01
2017-03-17 16:49:53.128 UTC [msp] Sign -> DEBU 009 Sign: digest: 3CEA3D149469841D625A8383FB0A50AD18F19D375502C79AABE5302804BE4365
Got status &{FORBIDDEN}
Error receiving: EOF
Error: EOF
MadhavaReddy (Fri, 17 Mar 2017 16:55:43 GMT):
root@72a84d8de556:/opt/gopath/src/github.com/hyperledger/fabric/peer# peer channel create -o orderer:7050 -c mychannel
2017-03-17 16:49:23.970 UTC [logging] InitFromViper -> DEBU 001 Setting default logging level to DEBUG for command 'channel'
2017-03-17 16:49:23.970 UTC [msp] GetLocalMSP -> DEBU 002 Returning existing local MSP
2017-03-17 16:49:23.970 UTC [msp] GetDefaultSigningIdentity -> DEBU 003 Obtaining default signing identity
Error connecting: rpc error: code = 14 desc = grpc: RPC failed fast due to transport failure
Error: rpc error: code = 14 desc = grpc: RPC failed fast due to transport failure
yacovm (Fri, 17 Mar 2017 17:08:58 GMT):
prefix your `peer channel create` with the MSP folder
yacovm (Fri, 17 Mar 2017 17:10:09 GMT):
`CORE_PEER_MSPCONFIGPATH=blablabla CORE_PEER_LOCALMSPID=yourMSPID peer channel create -o orderer0:7050 -c mychannel -f crypto/orderer/channel.tx`
MadhavaReddy (Fri, 17 Mar 2017 17:10:11 GMT):
i tried that also it was giving similar error
yacovm (Fri, 17 Mar 2017 17:10:23 GMT):
Also another tip
MadhavaReddy (Fri, 17 Mar 2017 17:10:24 GMT):
CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peer/peer0/localMspConfig CORE_PEER_ADDRESS=peer0:7051 CORE_PEER_LOCALMSPID="Org0MSP" peer channel create -o orderer0:7050 -c mychannel -f crypto/orderer/channel.tx
yacovm (Fri, 17 Mar 2017 17:10:35 GMT):
if you get a status FORBIDDEN
yacovm (Fri, 17 Mar 2017 17:10:40 GMT):
look at the orderer logs
MadhavaReddy (Fri, 17 Mar 2017 17:10:56 GMT):
ok
yacovm (Fri, 17 Mar 2017 17:10:58 GMT):
wait, are you inside docker, or outside?
yacovm (Fri, 17 Mar 2017 17:11:05 GMT):
peer0 works only inside docker
MadhavaReddy (Fri, 17 Mar 2017 17:12:05 GMT):
am inside cli container
yacovm (Fri, 17 Mar 2017 17:12:27 GMT):
ok
yacovm (Fri, 17 Mar 2017 17:12:36 GMT):
it should work
yacovm (Fri, 17 Mar 2017 17:13:04 GMT):
does the e2e_cli/network_setup script run successfully?
MadhavaReddy (Fri, 17 Mar 2017 17:13:55 GMT):
i ran only "generateCfgTrx.sh"
MadhavaReddy (Fri, 17 Mar 2017 17:14:00 GMT):
script
MadhavaReddy (Fri, 17 Mar 2017 17:14:36 GMT):
and commented "command: /bin/bash -c './scripts/script.sh ${CHANNEL_NAME}; '" this inside yaml file
MadhavaReddy (Fri, 17 Mar 2017 17:15:59 GMT):
root@72a84d8de556:/opt/gopath/src/github.com/hyperledger/fabric/peer# CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peer/peer0/localMspConfig CORE_PEER_ADDRESS=peer0:7051 CORE_PEER_LOCALMSPID="Org0MSP" peer channel create -o orderer0:7050 -c mychannel -f crypto/orderer/channel.tx
2017-03-17 17:15:29.287 UTC [logging] InitFromViper -> DEBU 001 Setting default logging level to DEBUG for command 'channel'
2017-03-17 17:15:29.287 UTC [msp] GetLocalMSP -> DEBU 002 Returning existing local MSP
2017-03-17 17:15:29.287 UTC [msp] GetDefaultSigningIdentity -> DEBU 003 Obtaining default signing identity
Error: Got unexpected status: BAD_REQUEST
Usage:
MadhavaReddy (Fri, 17 Mar 2017 17:16:36 GMT):
with configpath and mspid am getting bad request error
yacovm (Fri, 17 Mar 2017 17:16:52 GMT):
so, @MadhavaReddy can you please continue this discussion in the #fabric channel? It's not something related to gossip :)
yacovm (Fri, 17 Mar 2017 17:17:31 GMT):
I suggest you first try running the `network_setup.sh up` in the e2e_cli
MadhavaReddy (Fri, 17 Mar 2017 17:17:51 GMT):
actually if i remove peer logging in yaml file am seeing below error in docker logs
MadhavaReddy (Fri, 17 Mar 2017 17:18:09 GMT):
2017-03-17 16:37:25.653 UTC [gossip/comm#-1] GossipStream -> ERRO 001 Authentication failed
2017-03-17 16:37:26.056 UTC [gossip/comm#-1] GossipStream -> ERRO 002 Authentication failed
2017-03-17 16:37:50.654 UTC [gossip/comm#-1] GossipStream -> ERRO 003 Authentication failed
2017-03-17 16:37:51.057 UTC [gossip/comm#-1] GossipStream -> ERRO 004 Authentication failed
MadhavaReddy (Fri, 17 Mar 2017 17:18:34 GMT):
this will only show in peer0
yacovm (Fri, 17 Mar 2017 17:18:34 GMT):
that's the least of your concerns, however
MadhavaReddy (Fri, 17 Mar 2017 17:18:47 GMT):
:-) fine then
MadhavaReddy (Fri, 17 Mar 2017 17:18:51 GMT):
thank u
yacovm (Fri, 17 Mar 2017 17:18:55 GMT):
np.
MadhavaReddy (Fri, 17 Mar 2017 17:19:49 GMT):
do i need to run both network_setup.sh up as well as generateCfgTrx.sh script? last question :-)
MadhavaReddy (Fri, 17 Mar 2017 17:20:05 GMT):
or only network_setup.sh
yacovm (Fri, 17 Mar 2017 17:26:57 GMT):
ah
yacovm (Fri, 17 Mar 2017 17:27:12 GMT):
@MadhavaReddy only `./network_setup.sh up`
MadhavaReddy (Fri, 17 Mar 2017 17:27:44 GMT):
thank you @yacovm will run it
MadhavaReddy (Fri, 17 Mar 2017 17:58:18 GMT):
@yacovm fyi the CLI container existed once network_setup.sh execution is completed even same with generateCfgTrx.sh script, that means its not allowing to create a new channel from cli
yacovm (Fri, 17 Mar 2017 17:59:27 GMT):
Why does that mean it?
yacovm (Fri, 17 Mar 2017 18:00:18 GMT):
Existed or exited?
rameshthoomu (Fri, 17 Mar 2017 19:47:10 GMT):
Has joined the channel.
akashmar (Fri, 17 Mar 2017 19:48:20 GMT):
Has joined the channel.
rameshthoomu (Fri, 17 Mar 2017 19:48:33 GMT):
@MadhavaReddy Please see this [ ](https://chat.hyperledger.org/channel/fabric?msg=WSCTkFgcmr9QmW73Q) @rameshthoomu
akashmar (Fri, 17 Mar 2017 19:49:39 GMT):
I'm getting this gossip error even though I've made the certificates of all peer orgs available on each peer:
```
2017-03-17 19:51:39.157 UTC [gossip/comm#-1] authenticateRemotePeer -> WARN 1dd Identity store rejected 10.13.218.182:56098 : Peer Identity [0a 08 70 65 65 72 4f 72 67 32 12 f8 05 2d 2d 2d 2d 2d 42 45 47 49 4e 20 2... etc] cannot be validated. No MSP found able to do that.
2017-03-17 19:51:39.157 UTC [gossip/comm#-1] GossipStream -> ERRO 1de Authentication failed
```
Is there something I missing? If I disable gossip security, then it completely disregards the MSP configuration and uses DEFAULT org instead.
yacovm (Fri, 17 Mar 2017 19:51:45 GMT):
Oh, dont use the ignore security thing
yacovm (Fri, 17 Mar 2017 19:51:50 GMT):
It doesnt work anymore
yacovm (Fri, 17 Mar 2017 19:52:06 GMT):
We should remove it :worried:
akashmar (Fri, 17 Mar 2017 19:52:20 GMT):
Yeah I figured that out after I went looking into the code. So I removed it
akashmar (Fri, 17 Mar 2017 19:52:31 GMT):
Now I'm stuck with the above error
yacovm (Fri, 17 Mar 2017 19:52:45 GMT):
The error you're getting is that you get a message signed by a peer from a dofferent org
yacovm (Fri, 17 Mar 2017 19:52:58 GMT):
And that peer does not share any channel with you
akashmar (Fri, 17 Mar 2017 19:53:18 GMT):
So gossip is only intended for peers within the same org?
yacovm (Fri, 17 Mar 2017 19:53:25 GMT):
Of course not
yacovm (Fri, 17 Mar 2017 19:53:29 GMT):
I was just typing
yacovm (Fri, 17 Mar 2017 19:53:39 GMT):
Its a configuration problem :wink:
yacovm (Fri, 17 Mar 2017 19:54:04 GMT):
Something isnt configured right
akashmar (Fri, 17 Mar 2017 19:54:15 GMT):
I had a feeling I was missing something. How do I configure it properly?
Rymd (Fri, 17 Mar 2017 19:54:37 GMT):
Has joined the channel.
yacovm (Fri, 17 Mar 2017 19:54:44 GMT):
Ahhh... Thats a bit hard to explain. Perhaps tell about your setup
yacovm (Fri, 17 Mar 2017 19:54:51 GMT):
How many peers, orgs etc
akashmar (Fri, 17 Mar 2017 19:55:11 GMT):
I have 4 peers in 4 different orgs and 1 orderer
akashmar (Fri, 17 Mar 2017 19:55:33 GMT):
I'm running each container on a separate host
yacovm (Fri, 17 Mar 2017 19:56:41 GMT):
Okay, how did you generate the certs?
akashmar (Fri, 17 Mar 2017 19:56:52 GMT):
using the cryptogen utility from the repo
akashmar (Fri, 17 Mar 2017 19:57:45 GMT):
this is the current environment for each peer:
```
environment:
- CORE_PEER_ADDRESS=${PEER_ADDRESS}
- CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
- CORE_LOGGING_LEVEL=DEBUG
- CORE_PEER_NETWORKID=${PEER_NAME}
- CORE_NEXT=true
- CORE_PEER_ENDORSER_ENABLED=true
- CORE_PEER_COMMITTER_ENABLED=true
- CORE_PEER_ID=${PEER_NAME}
- CORE_PEER_PROFILE_ENABLED=false
- CORE_PEER_COMMITTER_LEDGER_ORDERER=${ORDERER_ADDRESS}
- CORE_PEER_GOSSIP_EXTERNALENDPOINT=${PEER_ADDRESS}
- CORE_PEER_LOCALMSPID=${MSP_ID}
- CORE_PBFT_GENERAL_N=4
- CORE_PEER_VALIDATOR_CONSENSUS_PLUGIN=pbft
- CORE_PBFT_GENERAL_TIMEOUT_REQUEST=10s
- CORE_PEER_CFG=/etc/hyperledger/fabric
- CORE_PEER_TLS_ENABLED=false
- CORE_PEER_GOSSIP_USELEADERELECTION=true
- CORE_PEER_GOSSIP_ORGLEADER=false
```
yacovm (Fri, 17 Mar 2017 19:57:52 GMT):
Cool
yacovm (Fri, 17 Mar 2017 19:57:55 GMT):
Nicr!
yacovm (Fri, 17 Mar 2017 19:57:55 GMT):
Nice
yacovm (Fri, 17 Mar 2017 19:58:43 GMT):
Hmmm
yacovm (Fri, 17 Mar 2017 19:58:51 GMT):
Where is local msp dir?
akashmar (Fri, 17 Mar 2017 19:59:22 GMT):
the default one is /etc/hyperledger/fabric/msp/sampleconfig. I'm just mounting the certificates there
akashmar (Fri, 17 Mar 2017 20:00:49 GMT):
here's the actual environment from docker container itself:
```
"CORE_PEER_ID=org1peer1",
"CORE_PEER_COMMITTER_LEDGER_ORDERER=10.13.218.180:7050",
"GOPATH=/opt/gopath",
"CORE_PEER_GOSSIP_USELEADERELECTION=true",
"CORE_PEER_COMMITTER_ENABLED=true",
"CORE_PBFT_GENERAL_N=4",
"CORE_PBFT_GENERAL_TIMEOUT_REQUEST=10s",
"CORE_PEER_ADDRESS=10.13.218.181:7051",
"CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock",
"CORE_PEER_CFG=/etc/hyperledger/fabric",
"CORE_PEER_TLS_ENABLED=false",
"CORE_PEER_GOSSIP_ORGLEADER=false",
"CORE_LOGGING_LEVEL=DEBUG",
"CORE_PEER_LOCALMSPID=peerOrg1",
"CORE_PEER_PROFILE_ENABLED=false",
"CORE_PEER_NETWORKID=org1peer1",
"CORE_PEER_GOSSIP_EXTERNALENDPOINT=10.13.218.181:7051",
"CORE_PEER_ENDORSER_ENABLED=true",
"CORE_NEXT=true",
"CORE_PEER_VALIDATOR_CONSENSUS_PLUGIN=pbft",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"PEER_CFG_PATH=/etc/hyperledger/fabric",
"CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/fabric/msp/sampleconfig"
```
saism (Fri, 17 Mar 2017 20:00:57 GMT):
Has joined the channel.
yacovm (Fri, 17 Mar 2017 20:01:22 GMT):
ah but it needs to be different for each peer!
yacovm (Fri, 17 Mar 2017 20:01:34 GMT):
are you somehow providing a different mount volume for each peer?
akashmar (Fri, 17 Mar 2017 20:01:38 GMT):
it is, this is the one from org1peer1
yacovm (Fri, 17 Mar 2017 20:02:09 GMT):
how do you ensure that peer0 and peer1 have different MSPCONFIGPATHs?
akashmar (Fri, 17 Mar 2017 20:02:32 GMT):
they're running on different hosts. they each get the MSPs from their own local dirs
yacovm (Fri, 17 Mar 2017 20:03:00 GMT):
but they're in docker
yacovm (Fri, 17 Mar 2017 20:03:01 GMT):
are they not
yacovm (Fri, 17 Mar 2017 20:03:03 GMT):
?
akashmar (Fri, 17 Mar 2017 20:03:53 GMT):
yes, but each container is running on a different host. I've copied the MSPs for each peer org to each host, and i'm mounting it there locally
yacovm (Fri, 17 Mar 2017 20:04:15 GMT):
ah got it
yacovm (Fri, 17 Mar 2017 20:04:32 GMT):
`"CORE_PEER_GOSSIP_EXTERNALENDPOINT=10.13.218.181:7051`
yacovm (Fri, 17 Mar 2017 20:04:35 GMT):
Nice!
yacovm (Fri, 17 Mar 2017 20:04:50 GMT):
did you read all the fabric code or something?
akashmar (Fri, 17 Mar 2017 20:05:17 GMT):
i'm getting there :)
yacovm (Fri, 17 Mar 2017 20:06:29 GMT):
okay
yacovm (Fri, 17 Mar 2017 20:06:31 GMT):
next question
yacovm (Fri, 17 Mar 2017 20:06:37 GMT):
have you created a channel
yacovm (Fri, 17 Mar 2017 20:06:40 GMT):
with the configtxgen
yacovm (Fri, 17 Mar 2017 20:06:41 GMT):
and all
akashmar (Fri, 17 Mar 2017 20:08:09 GMT):
not yet, this happens when I start the peers. I've tried creating a channel and that is successful, but I can't join it. I keep saying error 'No such channel' in the peer logs after I join
yacovm (Fri, 17 Mar 2017 20:08:34 GMT):
ahhhh
yacovm (Fri, 17 Mar 2017 20:08:37 GMT):
great!
yacovm (Fri, 17 Mar 2017 20:08:52 GMT):
so basically
yacovm (Fri, 17 Mar 2017 20:09:02 GMT):
you didn't create a channel?
yacovm (Fri, 17 Mar 2017 20:09:43 GMT):
I don't understand
yacovm (Fri, 17 Mar 2017 20:09:50 GMT):
you created a channel but haven't joined the peers?
akashmar (Fri, 17 Mar 2017 20:11:30 GMT):
i can create the channel but I haven't joined the peers
yacovm (Fri, 17 Mar 2017 20:11:36 GMT):
ah
yacovm (Fri, 17 Mar 2017 20:11:38 GMT):
that's great
yacovm (Fri, 17 Mar 2017 20:11:43 GMT):
it means my code works
yacovm (Fri, 17 Mar 2017 20:11:45 GMT):
so here is the deal
yacovm (Fri, 17 Mar 2017 20:12:03 GMT):
peers in different orgs *are not supposed to be able to communicate unless they share a channel*
akashmar (Fri, 17 Mar 2017 20:13:39 GMT):
I just created the channel and joined the peers and the errors disappeared :)
akashmar (Fri, 17 Mar 2017 20:13:54 GMT):
which confirms your point
yacovm (Fri, 17 Mar 2017 20:15:01 GMT):
I can't believe you doubted me @akashmar :O
akashmar (Fri, 17 Mar 2017 20:15:19 GMT):
I never did, I was trying that while you were typing :)
saism (Fri, 17 Mar 2017 20:16:02 GMT):
That was actually very helpful little discussion
akashmar (Fri, 17 Mar 2017 20:16:03 GMT):
that rule you mentioned, is that documented somewhere?
yacovm (Fri, 17 Mar 2017 20:17:38 GMT):
Hmm, let me see
yacovm (Fri, 17 Mar 2017 20:18:35 GMT):
well not really because I don't think we have a _ public _ document that describes *both gossip and MSP*
yacovm (Fri, 17 Mar 2017 20:18:45 GMT):
We are working on such a doc however!
yacovm (Fri, 17 Mar 2017 20:19:01 GMT):
But it's an architectural security constraint
akashmar (Fri, 17 Mar 2017 20:19:13 GMT):
I see
yacovm (Fri, 17 Mar 2017 20:19:14 GMT):
you don't want your peers to be able to talk to any organization, unless there is a reason to, right?
akashmar (Fri, 17 Mar 2017 20:19:37 GMT):
You're right, it makes sense
akashmar (Fri, 17 Mar 2017 20:20:17 GMT):
Are you able to help me with chaincode deployment, I have another problem there
yacovm (Fri, 17 Mar 2017 20:21:04 GMT):
I don't think I can because I'm not a chaincode developer but I can give it a shot, but in any case please do *not* ask chaincode questions in this channel :)
yacovm (Fri, 17 Mar 2017 20:21:13 GMT):
Ask in some other channel and tag me
yacovm (Fri, 17 Mar 2017 20:21:29 GMT):
but if it's something complicated I don't believe I would be able to
akashmar (Fri, 17 Mar 2017 20:22:09 GMT):
is there a channel specifically for chaincode?
yacovm (Fri, 17 Mar 2017 20:22:36 GMT):
I don't think so. Ask in #fabric or in #fabric-questions
GeorgeSamman (Mon, 20 Mar 2017 03:51:38 GMT):
Has joined the channel.
gennadyl (Mon, 20 Mar 2017 15:04:01 GMT):
Question about message store expiration - do we want to configure expiration timeout using yaml for each type of message?
yacovm (Mon, 20 Mar 2017 15:07:21 GMT):
I think not. It's best to make the period a function of some other duration
Xiao (Tue, 21 Mar 2017 08:08:04 GMT):
Has joined the channel.
thojest (Tue, 21 Mar 2017 10:22:30 GMT):
Has joined the channel.
thojest (Tue, 21 Mar 2017 10:23:35 GMT):
hey guys i have a question... im still using v0.6
thojest (Tue, 21 Mar 2017 10:23:54 GMT):
Is it possible to have a network of peer nodes on laptop 1 in docker containers
thojest (Tue, 21 Mar 2017 10:24:12 GMT):
and then connect a node on ANOTHER laptop to this existing network?
thojest (Tue, 21 Mar 2017 10:32:18 GMT):
i try to explain where i see the problem
thojest (Tue, 21 Mar 2017 10:32:42 GMT):
so suppose on laptop 1 we have a network of 4 peer nodes including the root node vp0
thojest (Tue, 21 Mar 2017 10:33:04 GMT):
now i try to connect from laptop2 another peer node lets call it vp4
thojest (Tue, 21 Mar 2017 10:33:24 GMT):
then vp4 finds vp0 BUT
thojest (Tue, 21 Mar 2017 10:33:37 GMT):
vp0 tells vp4 now the ip addresses of the docker containers on laptop1
thojest (Tue, 21 Mar 2017 10:33:55 GMT):
BUT vp4 thinks these ip addresses are on the network
thojest (Tue, 21 Mar 2017 10:34:18 GMT):
thus vp4 only gets the connection to vp0 the rootnode but it cannot communicate to vp1-3
yacovm (Tue, 21 Mar 2017 10:38:28 GMT):
So first of all- this channel is dedicated to the gossip component which is only in V1.0
Second- yes it's possible. You need to configure port forwarding and also set this address to the external address of the laptop: https://github.com/hyperledger/fabric/blob/v0.6/peer/core.yaml#L93
thojest (Tue, 21 Mar 2017 10:40:41 GMT):
sry didnt know about the restrictions of this channel but thanks
yacovm (Tue, 21 Mar 2017 10:41:03 GMT):
np
thojest (Tue, 21 Mar 2017 10:57:04 GMT):
@yacovm hmm getting the same problem
thojest (Tue, 21 Mar 2017 10:57:24 GMT):
in which channel should i ask this question?
yacovm (Tue, 21 Mar 2017 11:10:31 GMT):
#fabric
mdevilliers (Tue, 21 Mar 2017 16:34:08 GMT):
Has joined the channel.
matanyahu (Thu, 23 Mar 2017 10:54:28 GMT):
Has joined the channel.
joe-alewine (Fri, 24 Mar 2017 13:58:17 GMT):
Has joined the channel.
kostas (Fri, 24 Mar 2017 16:14:27 GMT):
Has joined the channel.
kostas (Fri, 24 Mar 2017 16:14:42 GMT):
Enlighten me a bit on this one:
kostas (Fri, 24 Mar 2017 16:14:56 GMT):
> However, to gossip a block to a peer will require that the ``/Channel/Application/Readers`` policy be satisfied.
kostas (Fri, 24 Mar 2017 16:15:05 GMT):
From `policies.rst`.
yacovm (Fri, 24 Mar 2017 16:15:29 GMT):
Hey Kostas how you doing?
kostas (Fri, 24 Mar 2017 16:15:35 GMT):
Why wouldn't we consult the `/Channel/Readers` policy for gossiping as well?
kostas (Fri, 24 Mar 2017 16:15:41 GMT):
Hey Yacov, doing well.
kostas (Fri, 24 Mar 2017 16:16:12 GMT):
I'm sure there's a good reason. `/Channel/Readers` only for those allowed to reach out to the ordering service directly?
kostas (Fri, 24 Mar 2017 16:16:34 GMT):
(But if that's the case... I have follow-up questions.)
yacovm (Fri, 24 Mar 2017 16:19:38 GMT):
Isn't channel readers for everyone including ordering service or something ?
yacovm (Fri, 24 Mar 2017 16:20:40 GMT):
I'm not an expert on this policies thing but I'd assume that the channel application readers is a more restrictive requirement isn't it?
kostas (Fri, 24 Mar 2017 16:25:07 GMT):
Don't quite know and I'm researching this now. There should be (and I'm sure there is) a good argument as to why we do both `Channel/Readers-Writers` and `Channel/Application/Readers-Writers`. Will keep on digging.
kostas (Fri, 24 Mar 2017 16:25:07 GMT):
Don't quite know and I'm researching this now. There should be (and I'm sure there is) a good argument as to why we do both `Channel/Readers-Writers` and `Channel/Application/Readers-Writers`. Off the top of my head though, I can't come up with a good, intuitive answer. Will keep on digging.
yacovm (Fri, 24 Mar 2017 16:25:47 GMT):
To be honest
yacovm (Fri, 24 Mar 2017 16:26:05 GMT):
We don't really consult these policies directly
yacovm (Fri, 24 Mar 2017 16:26:13 GMT):
we invoke a function that the MSP layer gives us
yacovm (Fri, 24 Mar 2017 16:26:21 GMT):
I'd ask in #fabric-crypto instead
kostas (Fri, 24 Mar 2017 20:02:23 GMT):
For my edification, do you plan to consult policies specific in the config eventually?
kostas (Fri, 24 Mar 2017 20:02:23 GMT):
For my edification, do you plan to consult policies specified in the config eventually?
kostas (Fri, 24 Mar 2017 20:02:45 GMT):
(Also, what is the function that you're using from the MSP layer?)
yacovm (Fri, 24 Mar 2017 20:07:47 GMT):
https://github.com/hyperledger/fabric/blob/master/gossip/api/crypto.go
implemented in https://github.com/hyperledger/fabric/blob/master/peer/gossip/mcs/mcs.go
yacovm (Fri, 24 Mar 2017 20:08:12 GMT):
And these policies stem from a config
nhrishi (Mon, 27 Mar 2017 13:06:59 GMT):
Has joined the channel.
manish-sethi (Fri, 31 Mar 2017 04:54:41 GMT):
@C0rWin, @yacovm - Can you guys have a look at https://gerrit.hyperledger.org/r/#/c/7303/.
This CR make some changes in ledger. However, one of the gossip tests is failing which appears totally unrelated to me.
```
03:18:22 unit-tests_1 | ? github.com/hyperledger/fabric/gossip/common [no test files]
03:18:44 unit-tests_1 | ok github.com/hyperledger/fabric/gossip/discovery 7.652s coverage: 83.2% of statements
03:19:06 unit-tests_1 | --- FAIL: TestPartition (3.10s)
03:19:06 unit-tests_1 | assertions.go:219:
Error Trace: election_test.go:330
03:19:06 unit-tests_1 |
Error: Should be true
03:19:06 unit-tests_1 |
Messages: Leadership callback result is wrong for p0
03:19:06 unit-tests_1 |
03:19:06 unit-tests_1 | FAIL
03:19:06 unit-tests_1 | coverage: 95.5% of statements
03:19:06 unit-tests_1 | FAIL github.com/hyperledger/fabric/gossip/election 7.166s
03:19:06 unit-tests_1 | error: exit status 1
03:19:06 unit-tests_1 | panic: EOF
03:19:06 unit-tests_1 |
03:19:06 unit-tests_1 | goroutine 1 [running]:
03:19:06 unit-tests_1 | panic(0x4daca0, 0xc420068020)
03:19:06 unit-tests_1 | /opt/go/src/runtime/panic.go:500 +0x1a1
03:19:06 unit-tests_1 | main.main()
03:19:06 unit-tests_1 | /opt/gotools/obj/gopath/src/github.com/AlekSi/gocov-xml/gocov-xml.go:60 +0x15fd
```
yacovm (Fri, 31 Mar 2017 08:18:04 GMT):
This is indeed unrelated, @manish-sethi.
It was supposed to be fixed in https://gerrit.hyperledger.org/r/#/c/7493/
But I pulled your patch set (2) and it appears that https://gerrit.hyperledger.org/r/#/c/7493/ didn't fix it... @C0rWin maybe we need to do that callback synchronous after all? I've never seen these failures before when it was changed to async.
Your build, however- failed again.
I opened a JIRA yesterday https://jira.hyperledger.org/browse/FAB-2944
yacovm (Fri, 31 Mar 2017 08:18:04 GMT):
This is indeed unrelated, @manish-sethi
It was supposed to be fixed in https://gerrit.hyperledger.org/r/#/c/7493/
But I pulled your patch set (2) and it appears that https://gerrit.hyperledger.org/r/#/c/7493/ didn't fix it... @C0rWin maybe we need to do that callback synchronous after all? I've never seen these failures before when it was changed to async.
Your build, however- failed again.
I opened a JIRA yesterday https://jira.hyperledger.org/browse/FAB-2944
yacovm (Fri, 31 Mar 2017 08:18:04 GMT):
This is indeed unrelated, @manish-sethi
It was supposed to be fixed in https://gerrit.hyperledger.org/r/#/c/7493/
But I pulled your patch set (2) and it appears that https://gerrit.hyperledger.org/r/#/c/7493/ didn't fix it... @C0rWin maybe we need to do that callback synchronous after all? I've never seen these failures before when it was changed to async. Or- maybe we were just wrong in the evaluation of the reason it fails.
Your build, however- failed again.
I opened a JIRA yesterday https://jira.hyperledger.org/browse/FAB-2944
manish-sethi (Fri, 31 Mar 2017 08:22:51 GMT):
@yacovm for me, all the tests had passed in my local vagrant. Yes, in the latest failure some of the other tests failed - though the last patch did not change anything other than fixing some indentation in proto file.
manish-sethi (Fri, 31 Mar 2017 08:24:04 GMT):
Let's see what happens in the latest build
choas (Sun, 02 Apr 2017 18:11:52 GMT):
Has joined the channel.
rahulhegde (Sun, 02 Apr 2017 20:56:44 GMT):
```
2017-04-02 20:54:52.004 UTC [gossip/election] IsLeader -> DEBU 99c [31 187 223 182 199 174 91 202 182 195 76 101 193 68 82 145 75 51 75 246 120 133 185 26 237 208 150 137 159 72 218 133] : Returning true
2017-04-02 20:54:52.004 UTC [gossip/gossip#172.25.0.2:7051] Gossip -> WARN 99d Failed obtaining gossipChannel of [109 121 99 104 97 110 110 101 108] aborting
2017-04-02 20:54:52.004 UTC [gossip/election] waitForInterrupt -> DEBU 99e [31 187 223 182 199 174 91 202 182 195 76 101 193 68 82 145 75 51 75 246 120 133 185 26 237 208 150 137 159 72 218 133] : Entering
```
May i know what is the significance of this warning and can be avoided?
yacovm (Sun, 02 Apr 2017 21:02:38 GMT):
this is a debug message
yacovm (Sun, 02 Apr 2017 21:03:02 GMT):
why would you want to avoid this?
rahulhegde (Sun, 02 Apr 2017 21:25:50 GMT):
I don't won't to avoid it but need to know what is wrong in the configuration to warn me?
rahulhegde (Sun, 02 Apr 2017 21:25:50 GMT):
I don't won't to avoid it but need to know what is wrong in the configuration to warn me? May be a documentation will also help @yacovm. Thanks.
rahulhegde (Sun, 02 Apr 2017 21:25:50 GMT):
I don't won't to avoid it but need to know what is wrong in the configuration to warn me? May be a documentation will also help @yacovm . Thanks.
rahulhegde (Sun, 02 Apr 2017 21:25:50 GMT):
I don't won't to avoid it but need to know what is wrong in the configuration to warn me? May be a documentation should be helpful, @yacovm . Thanks.
bobbiejc (Mon, 03 Apr 2017 15:56:39 GMT):
Has joined the channel.
bobbiejc (Mon, 03 Apr 2017 15:57:20 GMT):
naive question. Does gossip communicate between peers using GRPC?
bobbiejc (Mon, 03 Apr 2017 15:59:13 GMT):
and .. is it involved in the dissemination of blocks from the ordering service?
bobbiejc (Mon, 03 Apr 2017 15:59:29 GMT):
@david.enyeart ^^
yacovm (Mon, 03 Apr 2017 16:17:02 GMT):
@bobbiejc yes. It uses the same gRPC server as the peer
yacovm (Mon, 03 Apr 2017 16:17:08 GMT):
and regarding the dissemination - correct
dave.enyeart (Mon, 03 Apr 2017 16:17:50 GMT):
for leader to non-leader peers Gossip is used. How about from ordering service to leaders? Is that still Gossip (over gRPC)?
yacovm (Mon, 03 Apr 2017 16:20:59 GMT):
no
yacovm (Mon, 03 Apr 2017 16:21:08 GMT):
that code is in `core/deliveryservice`
yacovm (Mon, 03 Apr 2017 16:21:29 GMT):
but... it sort-of goes through gossip
yacovm (Mon, 03 Apr 2017 16:21:46 GMT):
I mean package wise
dave.enyeart (Mon, 03 Apr 2017 16:21:49 GMT):
Bobbie is trying to explain to some newbies (that dont know the code). so conceptually can you explain sort-of
yacovm (Mon, 03 Apr 2017 16:21:51 GMT):
not communication wise
yacovm (Mon, 03 Apr 2017 16:21:53 GMT):
ah
yacovm (Mon, 03 Apr 2017 16:21:55 GMT):
so don't mention it
yacovm (Mon, 03 Apr 2017 16:22:11 GMT):
depends on whether you want to know the code, or want to know the big picture
dave.enyeart (Mon, 03 Apr 2017 16:22:35 GMT):
so from big picture, we can just say that leader peers get blocks from ordering service via grpc?
yacovm (Mon, 03 Apr 2017 16:23:45 GMT):
yeah directly via gRPC by calling `Deliver` on the ordering service
dave.enyeart (Mon, 03 Apr 2017 16:24:07 GMT):
ok cool. and furthermore dissemination between leader peers and non-leaders is via gossip
dave.enyeart (Mon, 03 Apr 2017 16:24:55 GMT):
is Gossip used anywhere besides state transfer and leader to non-leader dissemination?
yacovm (Mon, 03 Apr 2017 16:25:20 GMT):
what do you mean anywhere
yacovm (Mon, 03 Apr 2017 16:25:46 GMT):
as a function for fabric?
dave.enyeart (Mon, 03 Apr 2017 16:25:52 GMT):
are these the only two functional interactions that gossis is involved in? 1) state transfer and 2 ) leader to non-leader dissemination
dave.enyeart (Mon, 03 Apr 2017 16:26:03 GMT):
right
yacovm (Mon, 03 Apr 2017 16:26:59 GMT):
yeah basically
dave.enyeart (Mon, 03 Apr 2017 16:27:07 GMT):
ok thx
yacovm (Mon, 03 Apr 2017 16:27:09 GMT):
np
bobbiejc (Mon, 03 Apr 2017 17:08:28 GMT):
thanks!
kostas (Mon, 03 Apr 2017 21:10:26 GMT):
Given this message in FAB-2949, am I right to assume that this functionality is not yet in?
> Peer to Ordering service: The configuration blocks come with a set of endpoints of ordering service nodes. We (Haifa team) are working on a set of change sets, that enable the peer to re-connect to different ordering service nodes out of the set defined in the configuration block, should the current ordering service node become unreachable.
kostas (Mon, 03 Apr 2017 21:10:26 GMT):
Given this message in [FAB-2949](https://jira.hyperledger.org/browse/FAB-2949), am I right to assume that this functionality is not yet in?
> Peer to Ordering service: The configuration blocks come with a set of endpoints of ordering service nodes. We (Haifa team) are working on a set of change sets, that enable the peer to re-connect to different ordering service nodes out of the set defined in the configuration block, should the current ordering service node become unreachable.
C0rWin (Mon, 03 Apr 2017 21:23:10 GMT):
@kostas there is still one working item in progress to finalize this
kostas (Mon, 03 Apr 2017 21:23:25 GMT):
@COrW
C0rWin (Mon, 03 Apr 2017 21:23:51 GMT):
there is no reconnection strategy to ordering service committed yet
kostas (Mon, 03 Apr 2017 21:24:04 GMT):
@C0rWin: Got it, thanks. Can you @ me and @mrshah-ibm when this is merged?
mrshah-ibm (Mon, 03 Apr 2017 21:24:05 GMT):
Has joined the channel.
C0rWin (Mon, 03 Apr 2017 21:24:19 GMT):
Sure
C0rWin (Mon, 03 Apr 2017 21:24:32 GMT):
I can provide you the JIRA item to track
kostas (Mon, 03 Apr 2017 21:24:39 GMT):
That'll be great.
kostas (Mon, 03 Apr 2017 21:27:30 GMT):
Also, I am dealing with an issue where the restarted peer cannot read blocks from a restarted orderer. Looking at the logs now, from your side what is the component I should be focusing on? I am guessing `deliveryClient` right?
yacovm (Mon, 03 Apr 2017 21:32:04 GMT):
yep
C0rWin (Mon, 03 Apr 2017 21:33:33 GMT):
@kostas https://jira.hyperledger.org/browse/FAB-2759
scottz (Wed, 05 Apr 2017 15:35:23 GMT):
I am running a test, with two orgs, create a channel, and attempt peers to join the channel but I see error in peer logs; gossip is saying: "Tried joining channel test1 but our org( DEFAULT ), isn't among the orgs of the channel: [PeerOrg1 PeerOrg2] , aborting." Can anyone suggest anything that might be configured wrong? Why is gossip on the peer using "DEFAULT" instead of PeerOrg1? I am trying to run this inside vagrant, using PTE and node sdk; a colleague does not see the error when running same test without using vagrant, so I think the certs are all correct, which is why I am confused.
scottz (Wed, 05 Apr 2017 15:37:03 GMT):
Immediately after, I see another log (I think the ascii translation of the bytes is "testOrg1"): [gossip/gossip#172.19.0.5:7051] UpdateChannelMetadata -> DEBU 2c0 No such channel [116 101 115 116 79 114 103 49]
scottz (Wed, 05 Apr 2017 15:37:03 GMT):
Immediately after, I see another log (I think the ascii translation of the bytes is "test1"): [gossip/gossip#172.19.0.5:7051] UpdateChannelMetadata -> DEBU 2c0 No such channel [116 101 115 116 49]
scottz (Wed, 05 Apr 2017 15:39:35 GMT):
... using git commit commit fa3d88cd Release 1.0.0-alpha
yacovm (Wed, 05 Apr 2017 17:20:46 GMT):
Hey @scottz .
This is a message that the gossip layer prints in order to notify to the operator that the peer's MSP-ID (DEFAULT) isn't one of the organizations in the configuration block.
yacovm (Wed, 05 Apr 2017 17:21:08 GMT):
I guess the peer has started with a wrong MSP ID config
yacovm (Wed, 05 Apr 2017 17:21:32 GMT):
the certs might be ok but the MSP_ID isn't derived from the certs, but from `core/peer.yaml`
yacovm (Wed, 05 Apr 2017 17:22:25 GMT):
now, the "No such channel" message you see- is that the gossip component received a metadata update invocation from the peer (upper layers) but since the "join channel" on the gossip failed, it doesn't know what to do with it
yacovm (Wed, 05 Apr 2017 17:23:16 GMT):
https://github.com/hyperledger/fabric/blob/master/peer/core.yaml#L240 this sets that value
scottz (Wed, 05 Apr 2017 18:08:22 GMT):
@yacovm In docker-compose.yml, for peer0 and peer1, we had set CORE_PEER_LOCALMSPID=Peer1MSP, which should override the value "DEFAULT" defined in fabric/peer/core.yaml.
yacovm (Wed, 05 Apr 2017 18:10:08 GMT):
Even so- why is it Peer1MSP and not as the log error wants you to? PeerOrg1?
scottz (Wed, 05 Apr 2017 19:34:56 GMT):
Peer1MSP is the ID of the organization. (DEFAULT is the chainID too.) PeerOrg1 is the Name.orgListFromConfig() returns the list of names.
yacovm (Wed, 05 Apr 2017 19:38:18 GMT):
They *have* to be the same in the configtx.yaml
scottz (Wed, 05 Apr 2017 21:02:17 GMT):
Ummm, that makes no sense. First, the code should not print an ID and compare it to names. Second: then why do we have two variables? We should eliminate one. Third: values are not the same for example in fabric/common/configtx/tool/configtx.yaml Organizations / &OrdererOrg1 (Name:OrdererOrg1, ID:Orderer1MSP)
yacovm (Wed, 05 Apr 2017 21:05:13 GMT):
> First, the code should not print an ID and compare it to names
Which code? The code that prints the log you described in gossip?
> Second: then why do we have two variables? We should eliminate one.
I have said that long ago, but I do not make the calls. And I guess you mean two config options, not variables, right?
> Third: values are not the same for example in fabric/common/configtx/tool/configtx.yaml
They should be the same for the peers, if you want things to work.
scottz (Wed, 05 Apr 2017 21:08:19 GMT):
yes. yes. yes, ok. I will create 3 bugs for these items, and a 4th bug for the original error I was asking about "Tried joining channel ...". Maybe one or more will not be fixed, but we will try.
yacovm (Wed, 05 Apr 2017 21:08:42 GMT):
wait
yacovm (Wed, 05 Apr 2017 21:08:47 GMT):
wait
yacovm (Wed, 05 Apr 2017 21:08:48 GMT):
3 bugs?
yacovm (Wed, 05 Apr 2017 21:08:55 GMT):
what's the first?
yacovm (Wed, 05 Apr 2017 21:09:14 GMT):
about the 4th one
yacovm (Wed, 05 Apr 2017 21:09:17 GMT):
How is this a bug?
yacovm (Wed, 05 Apr 2017 21:09:30 GMT):
This is a warning that I put in the code, on request of @bmos299
bmos299 (Wed, 05 Apr 2017 21:09:30 GMT):
Has joined the channel.
yacovm (Wed, 05 Apr 2017 21:09:41 GMT):
where is the bug exactly?
C0rWin (Wed, 05 Apr 2017 21:15:07 GMT):
@scottz hi, can you please explain, how misconfiguration turned to be the bug?
C0rWin (Wed, 05 Apr 2017 21:15:25 GMT):
I mean there is clearly wrong configuration provided
C0rWin (Wed, 05 Apr 2017 21:15:31 GMT):
you've been alerted about it
C0rWin (Wed, 05 Apr 2017 21:15:35 GMT):
what is the bug here?
C0rWin (Wed, 05 Apr 2017 21:16:27 GMT):
So I understand that misconfiguration itself is a bug, but you just need to fix and provide a correct one
C0rWin (Wed, 05 Apr 2017 21:16:45 GMT):
or you would like to be alerted in different way?
C0rWin (Wed, 05 Apr 2017 21:16:55 GMT):
also how this gossip related?
scottz (Wed, 05 Apr 2017 21:17:39 GMT):
1. bug is for inconsistency in the log: "our org ( ID ) isn't among ( list of Org Names )".
yacovm (Wed, 05 Apr 2017 21:17:54 GMT):
This is not an inconsistency
yacovm (Wed, 05 Apr 2017 21:18:02 GMT):
this is on purpose
yacovm (Wed, 05 Apr 2017 21:18:07 GMT):
it prints the name of your org
yacovm (Wed, 05 Apr 2017 21:18:13 GMT):
and the names of the orgs in the block
scottz (Wed, 05 Apr 2017 21:18:18 GMT):
no it does not.
scottz (Wed, 05 Apr 2017 21:18:22 GMT):
it prints the ID of the org
scottz (Wed, 05 Apr 2017 21:18:36 GMT):
and the names of the orgs in the block.
scottz (Wed, 05 Apr 2017 21:19:25 GMT):
(2.) The whole thing about the name and ID being required to be the same ... I will create a different bug for that one.
yacovm (Wed, 05 Apr 2017 21:19:36 GMT):
exactly!
yacovm (Wed, 05 Apr 2017 21:19:43 GMT):
that's what its supposed to print :)
scottz (Wed, 05 Apr 2017 21:22:15 GMT):
why don't you just print both the org name AND org ID and state that the org name and ID must match for gossip to work, plus it must be in the list of possible org names
scottz (Wed, 05 Apr 2017 21:22:43 GMT):
... if indeed that is a requirement - and I have not heard a reason WHY that must be so.
yacovm (Wed, 05 Apr 2017 21:23:02 GMT):
there is no such thing as org name
yacovm (Wed, 05 Apr 2017 21:23:37 GMT):
to the peer there is only org ID
yacovm (Wed, 05 Apr 2017 21:23:50 GMT):
as for the rest
yacovm (Wed, 05 Apr 2017 21:24:31 GMT):
it's up to the operator of the environment to take care that the org ID of the peer matches one of the orgs in the config block
yacovm (Wed, 05 Apr 2017 21:25:01 GMT):
> ... if indeed that is a requirement - and I have not heard a reason WHY that must be so.
OK, this is a different problem, still not a bug though. maybe a JIRA story at best
C0rWin (Wed, 05 Apr 2017 21:27:36 GMT):
> ... if indeed that is a requirement - and I have not heard a reason WHY that must be so.
^^^ This is very explicit requirement, since you need to make sure peer belongs to the org and eligible to participate
scottz (Wed, 05 Apr 2017 21:30:51 GMT):
Profiles:
# SampleSingleMSPSolo defines a configuration which contains a single MSP
# definition (the MSP sampleconfig).
testOrg:
Orderer:
<<: *OrdererDefaults
Organizations:
- *OrdererOrg1
Application:
<<: *ApplicationDefaults
Organizations:
- *PeerOrg1
- *PeerOrg2
Organizations:
- &PeerOrg1
Name: PeerOrg1
ID: Peer1MSP
yacovm (Wed, 05 Apr 2017 21:31:30 GMT):
So Name and ID needs to be the same
yacovm (Wed, 05 Apr 2017 21:31:30 GMT):
So Name and ID need to be the same
scottz (Wed, 05 Apr 2017 21:31:30 GMT):
As you can see from this snippet of a configtx.yaml file, the PeerOrg contains both a Name and an ID. Are you saying they need to be the same?
C0rWin (Wed, 05 Apr 2017 21:32:16 GMT):
yes
scottz (Wed, 05 Apr 2017 21:35:30 GMT):
Why isn't that explained in this file configtx.yaml?
scottz (Wed, 05 Apr 2017 21:35:52 GMT):
An MSP ID is not the same as an Org name.
yacovm (Wed, 05 Apr 2017 21:36:18 GMT):
Org name, is just a friendly representation of an organization
yacovm (Wed, 05 Apr 2017 21:36:27 GMT):
but in practice it needs to be the same
yacovm (Wed, 05 Apr 2017 21:36:33 GMT):
in order for stuff to work
yacovm (Wed, 05 Apr 2017 21:36:43 GMT):
this was discussed
yacovm (Wed, 05 Apr 2017 21:36:59 GMT):
before the alpha version was released
yacovm (Wed, 05 Apr 2017 21:37:22 GMT):
regarding why it's not explained- maybe you have a point there. You can open a bug for that I guess
scottz (Wed, 05 Apr 2017 21:38:24 GMT):
ok.
scottz (Wed, 05 Apr 2017 21:40:48 GMT):
And I still say your code illegally compares an ID with Names (even if they are "suppposed to be the same"), and it would help if you printed both the name and the ID in this log. Can you agree to that?
yacovm (Wed, 05 Apr 2017 21:41:13 GMT):
No, because I only have access to one of them
yacovm (Wed, 05 Apr 2017 21:41:19 GMT):
That's the API we were given
yacovm (Wed, 05 Apr 2017 21:41:29 GMT):
I work with what I have
yacovm (Wed, 05 Apr 2017 21:46:58 GMT):
https://github.com/hyperledger/fabric/blob/master/common/config/api.go#L48
This ^ is the API that needs to be changed in order to support what you want
yacovm (Wed, 05 Apr 2017 21:48:17 GMT):
You're basically saying- why are 2 configuration attributes are in the configtx.yaml file, but they need to be the same?
I agree, but the people that are in charge of the config, and on the API above that I put in the link are roaming in the #fabric-consensus channel :)
yacovm (Wed, 05 Apr 2017 21:48:36 GMT):
I can not do anything about that, as I only consume an API that I was given.
scottz (Wed, 05 Apr 2017 21:52:59 GMT):
How can they have anything to say about the PEER's Name and ID config parameters? Or are you saying they must match within an Orderer org too?
yacovm (Wed, 05 Apr 2017 21:53:22 GMT):
it has nothing to do with a peer's name
yacovm (Wed, 05 Apr 2017 21:53:43 GMT):
a peer belongs to some organization
yacovm (Wed, 05 Apr 2017 21:53:51 GMT):
the organization is identified by the MSP id
yacovm (Wed, 05 Apr 2017 21:55:36 GMT):
Having said that, @scottz I'm off for the rest of the night.
If you have deeper questions about the configuration I suggest you ask in #fabric-consensus as they are responsible for that.
If you have further questions about gossip, you are free to ask here.
yacovm (Wed, 05 Apr 2017 21:56:11 GMT):
(But I'm going now so... this would have to wait for tomorrow :) )
yacovm (Wed, 05 Apr 2017 21:56:11 GMT):
(But I'm going now so... this would have to wait for tomorrow :) or if C0rwin is still awake)
scottz (Wed, 05 Apr 2017 21:57:06 GMT):
ok
scottz (Wed, 05 Apr 2017 22:52:43 GMT):
@yacovm you could have mentioned that you already opened FAB-2499
yacovm (Wed, 05 Apr 2017 22:53:26 GMT):
Have you already opened another one?
C0rWin (Wed, 05 Apr 2017 23:31:38 GMT):
I guess this was not very clear what exactly you are trying to claim
C0rWin (Wed, 05 Apr 2017 23:32:14 GMT):
as you stated there is something wrong from gossip functionality perpective
C0rWin (Wed, 05 Apr 2017 23:32:34 GMT):
and clear gossip acted as expected
AmberZhang (Thu, 06 Apr 2017 07:16:15 GMT):
Has joined the channel.
lizhih (Thu, 06 Apr 2017 09:43:55 GMT):
Has joined the channel.
scottz (Thu, 06 Apr 2017 16:21:15 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=neqQ3pPjWFvTrAyPK) @yacovm Actually, Organizations returns ApplicationOrg, which contains Org, which contains method MSPID() in addition to Name(). Maybe that method did not exist before, but it seems to me that you can add a bit of code to orgListFromConfig() that digs a bit deeper for the MSPID instead of just taking the Name? Then you would be comparing ID to a list of IDs, and you would not be forcing user to set ID equal to Name.
yacovm (Thu, 06 Apr 2017 16:34:06 GMT):
Yeah, @scottz , nice observation! :) @C0rWin what do you think?
scottz (Thu, 06 Apr 2017 16:50:34 GMT):
thanks. I agree that seems to be the best solution, and would clean up all the items I mentioned. hopefully that will work.
scottz (Fri, 07 Apr 2017 13:33:02 GMT):
@yacovm @C0rWin can you guys add a note to FAB-2499 (and maybe change the title), and take ownership of it, to get that done? if so, then I won't create any more bugs/issues.
C0rWin (Fri, 07 Apr 2017 13:34:37 GMT):
This has nothing to do with gossip, please ask #fabric-consensus
C0rWin (Fri, 07 Apr 2017 13:34:50 GMT):
They own the API
yacovm (Fri, 07 Apr 2017 13:35:07 GMT):
I think he pointed out a way of not changing the API though
yacovm (Fri, 07 Apr 2017 13:35:17 GMT):
Scott, we'll take a closer look next week
yacovm (Fri, 07 Apr 2017 13:35:21 GMT):
and discuss it internally
scottz (Fri, 07 Apr 2017 13:38:29 GMT):
ok. let me know next week. thanks.
mychewcents (Sun, 09 Apr 2017 12:09:05 GMT):
Has left the channel.
zian (Mon, 10 Apr 2017 13:58:34 GMT):
Has joined the channel.
jeroiraz (Mon, 10 Apr 2017 18:40:30 GMT):
Has joined the channel.
Ratnakar (Tue, 11 Apr 2017 13:41:40 GMT):
Has joined the channel.
Ratnakar (Tue, 11 Apr 2017 16:47:48 GMT):
@yacovm In a two peer organization when CORE_PEER_GOSSIP_ORGLEADER=true for an anchor peer in each organization, while CORE_PEER_GOSSIP_ORGLEADER=false only on non-anchor peers, then processing transactions on those non-anchor peers fails with following error
```
failed to obtain cds for end2end - transaction not found end2end/mychannel
```
Is this a valid configuation ? If not, then why not ?
Ratnakar (Tue, 11 Apr 2017 16:47:48 GMT):
@yacovm @C0rWin , In a two peer organization when CORE_PEER_GOSSIP_ORGLEADER=true for an anchor peer in each organization, while CORE_PEER_GOSSIP_ORGLEADER=false only on non-anchor peers, then processing transactions on those non-anchor peers fails with following error
```
failed to obtain cds for end2end - transaction not found end2end/mychannel
```
Is this a valid configuation ? If not, then why not ?
yacovm (Tue, 11 Apr 2017 16:50:34 GMT):
you have a peer in one of the organizations that is set to ORGLEADER=true and the other peer shows you this error?
yacovm (Tue, 11 Apr 2017 16:54:11 GMT):
Anyway- I just checked:
I modified and set useleaderelection to false in the peer-base and then turned on orgleader on the anchor peers, and ran the e2e test and it passed
yacovm (Tue, 11 Apr 2017 16:54:24 GMT):
so "works for me" :)
Ratnakar (Tue, 11 Apr 2017 18:14:53 GMT):
@yacovm My bad , I should have tried e2e_cli example too , I was using docker-compose file from node-sdk repo https://github.com/hyperledger/fabric-sdk-node/blob/master/test/fixtures/docker-compose.yaml
The difference I see between thes two docker-compose files is that , In e2e_cli example we have `CORE_PEER_GOSSIP_SKIPHANDSHAKE=true`
yacovm (Tue, 11 Apr 2017 18:15:15 GMT):
yep. that should be there too.
yacovm (Tue, 11 Apr 2017 18:15:23 GMT):
but you could've also looked at the logs
yacovm (Tue, 11 Apr 2017 18:15:29 GMT):
gossip complains when this option isn't set
yacovm (Tue, 11 Apr 2017 18:15:33 GMT):
and logs a WARN/ERR
Ratnakar (Tue, 11 Apr 2017 18:16:59 GMT):
I disabled DEBUG as i see too much logging when I ran some tests ...
Ratnakar (Tue, 11 Apr 2017 18:17:21 GMT):
Thankyou for point it I shall check it out
yacovm (Tue, 11 Apr 2017 18:18:25 GMT):
I don't think it's in debug
Ratnakar (Tue, 11 Apr 2017 18:19:32 GMT):
So I understand when TLS is enabled this flag is must be set, but I would also like to know why we don't see any issue when we send transactions on `Anchor peers`.
Ratnakar (Tue, 11 Apr 2017 18:19:32 GMT):
So now I understand when TLS is enabled this flag is must be set `CORE_PEER_GOSSIP_SKIPHANDSHAKE=true`, but I would also like to know why we don't see any issue when we send transactions on `Anchor peers`.
Ratnakar (Tue, 11 Apr 2017 18:19:39 GMT):
Can you please explain
yacovm (Tue, 11 Apr 2017 18:20:41 GMT):
I don't understand what you're asking
yacovm (Tue, 11 Apr 2017 18:20:44 GMT):
what transactions
Ratnakar (Tue, 11 Apr 2017 18:21:21 GMT):
Invokes/Query on anchor peer works but fails on non-anchor peers
yacovm (Tue, 11 Apr 2017 18:22:43 GMT):
ah because they can't communicate because of TLS expects mutual-TLS, unless SKIPHANDSHAKE is true
yacovm (Tue, 11 Apr 2017 18:22:47 GMT):
we don't have mutual TLS yet
yacovm (Tue, 11 Apr 2017 18:22:52 GMT):
so we use on-way TLS
yacovm (Tue, 11 Apr 2017 18:23:14 GMT):
since you made one non-leader and one-leader, if they can't communicate... they can't send each other blocks
Ratnakar (Tue, 11 Apr 2017 18:35:42 GMT):
Thankyou @yacovm that explains all :)
yacovm (Tue, 11 Apr 2017 18:36:07 GMT):
np
Lin-YiTang (Tue, 11 Apr 2017 21:20:04 GMT):
Has joined the channel.
subbu165 (Wed, 12 Apr 2017 11:03:05 GMT):
Has joined the channel.
subbu165 (Wed, 12 Apr 2017 11:03:39 GMT):
Has left the channel.
harrijk (Wed, 12 Apr 2017 17:37:41 GMT):
Can peers only perform a gossip bootstrap within their own organization?
yacovm (Wed, 12 Apr 2017 18:03:10 GMT):
The bootstrap should only contain such
harrijk (Wed, 12 Apr 2017 18:11:02 GMT):
Does it only make sense to set a gossip external endpoint on a peer that has been designated an anchor peer as defined in a configtx.yaml file?
yacovm (Wed, 12 Apr 2017 18:14:35 GMT):
So- I'd say, anchor peers *should have* external endpoints
yacovm (Wed, 12 Apr 2017 18:14:44 GMT):
for other peers- depends on installation.
yacovm (Wed, 12 Apr 2017 18:15:26 GMT):
if you think a peer has an external endpoint (routable from other orgs) and you have a mechanism for resurrecting it, in case it dies, I'd advise simply make it an anchor peer
yacovm (Wed, 12 Apr 2017 18:15:32 GMT):
you can have many anchor peers per org
troyronda (Thu, 13 Apr 2017 00:05:36 GMT):
I'm wondering what mechanism gossip is using to ensure that an enrollment certificate belong to a peer and not a client? Is this somewhere in a config txn or in the cert or other?
troyronda (Thu, 13 Apr 2017 00:05:36 GMT):
I'm wondering what mechanism gossip is using to ensure that an enrollment certificate belongs to a peer and not a client? Is this somewhere in a config txn or in the cert or other?
yacovm (Thu, 13 Apr 2017 06:03:09 GMT):
@troyronda clients don't have the code to speak with peers. I'm not sure what is your concern here
yacovm (Thu, 13 Apr 2017 06:04:17 GMT):
You should be worried from something else, I'd say- that someone takes a peer, and gives it a client certificate, and that will be validated. No?
troyronda (Thu, 13 Apr 2017 10:56:58 GMT):
I don't understand - they can download the peer code from github :)
troyronda (Thu, 13 Apr 2017 10:58:05 GMT):
My concern is that a client who got a certificate and with that is allowed to pretend to be a peer, they could intercept all the gossip traffic
troyronda (Thu, 13 Apr 2017 10:58:30 GMT):
This must not be allowed especially since clients should not see all the data in the ledger network.
troyronda (Thu, 13 Apr 2017 10:58:30 GMT):
This must not be allowed especially since clients must not see all the data in the ledger network.
troyronda (Thu, 13 Apr 2017 11:10:03 GMT):
If you are implying that not having user chaincode takes away the attack, this is not the case -- the blocks and gossip traffic can have sensitive data that doesn't require code to interpret.
yacovm (Thu, 13 Apr 2017 11:16:02 GMT):
user chaincode?
yacovm (Thu, 13 Apr 2017 11:16:33 GMT):
look
yacovm (Thu, 13 Apr 2017 11:16:36 GMT):
I'll exlpain
yacovm (Thu, 13 Apr 2017 11:17:13 GMT):
In gossip, in order for 2 peers to communicate they need to perform a secure handshake
yacovm (Thu, 13 Apr 2017 11:17:56 GMT):
the secure handshake uses the signing identity of the peer (meaning- the peer needs a private key, and a peer's identity)
yacovm (Thu, 13 Apr 2017 11:19:50 GMT):
Now... if from some reason, a client's certificate and a client's private can be used for that - that's a problem but there is nothing that can be done in the gossip code for that since we just use the MSP infrastructure
troyronda (Thu, 13 Apr 2017 11:20:42 GMT):
ok - i mentioned the chaincode because that's the code the client wouldn't have. the rest of the peer code, they can obtain.
troyronda (Thu, 13 Apr 2017 11:20:51 GMT):
who should i ask regarding the MSP infrastrucuture
yacovm (Thu, 13 Apr 2017 11:22:29 GMT):
@elli-androulaki
elli-androulaki (Thu, 13 Apr 2017 11:22:29 GMT):
Has joined the channel.
yacovm (Thu, 13 Apr 2017 11:23:38 GMT):
The question you need to ask is - is there a way to differentiate from a client's cert to a peer's cert?
troyronda (Thu, 13 Apr 2017 11:24:01 GMT):
@smithbk ^^
smithbk (Thu, 13 Apr 2017 11:24:01 GMT):
Has joined the channel.
yacovm (Thu, 13 Apr 2017 11:24:04 GMT):
it also depends on the MSP types that you have in the system
yacovm (Thu, 13 Apr 2017 11:24:50 GMT):
if for example you have both an MSP of type X and an MSP of type Y, and the PKI- infrastructure of the "network" only provides identities of type X to clients, and identities of type Y to peers, then you're safe, no?
troyronda (Thu, 13 Apr 2017 11:27:28 GMT):
Yeh.. but there are ongoing discussions on restricting eventHub to needing clients to be on the same MSP as the peer in 1.0 ( @binhn)
binhn (Thu, 13 Apr 2017 11:27:28 GMT):
Has joined the channel.
yacovm (Thu, 13 Apr 2017 11:28:47 GMT):
I am aware of these discussions
yacovm (Thu, 13 Apr 2017 11:28:57 GMT):
But I still don't see your point regarding gossip
yacovm (Thu, 13 Apr 2017 11:29:05 GMT):
aha I see
yacovm (Thu, 13 Apr 2017 11:29:09 GMT):
You're saying the following:
yacovm (Thu, 13 Apr 2017 11:29:48 GMT):
since a client can connect to a peer, it means its certificate is (perhaps) likely to be validated by the peer's MSP - and that means that a client's certificate can be used as a peer certficate
yacovm (Thu, 13 Apr 2017 11:29:50 GMT):
right?
troyronda (Thu, 13 Apr 2017 11:30:16 GMT):
yes
yacovm (Thu, 13 Apr 2017 11:31:55 GMT):
Well- I'd say it's a sound concern, but it's not a gossip issue at all :)
We just blindly invoke methods that the MSP layer provides. If the MSP layer can make it so a client's certificate can not be validated by the implementation of the API between gossip and the MSP, then you're safe.
troyronda (Thu, 13 Apr 2017 11:32:56 GMT):
i hear ya - i was just browsing the gossip code to see if there was something additional (didn't find so asked) :)
yacovm (Thu, 13 Apr 2017 11:33:18 GMT):
understood.
yacovm (Thu, 13 Apr 2017 11:34:58 GMT):
The API I'm talking about, @troyronda is in https://github.com/hyperledger/fabric/blob/master/gossip/api/crypto.go btw
elli-androulaki (Thu, 13 Apr 2017 14:39:59 GMT):
Right, essentially currently peers implicitly include in their "trustzone" nodes that are members of the same msp as the peer (have valid identities under the same MSP). The underlying assumption was here that msps map to orgs in a 1-1 fashion and that clients and peers do not need to be distinguished.
troyronda (Thu, 13 Apr 2017 14:52:29 GMT):
Clients must not be trusted at the same level as peers
vpaprots (Thu, 13 Apr 2017 17:09:11 GMT):
Has joined the channel.
LordGoodman (Tue, 18 Apr 2017 08:09:42 GMT):
Has joined the channel.
Gaurav_Impro (Wed, 19 Apr 2017 09:53:11 GMT):
Has joined the channel.
ioctl (Sat, 22 Apr 2017 16:34:29 GMT):
Has joined the channel.
Willson (Tue, 25 Apr 2017 14:27:43 GMT):
Has joined the channel.
joe-alewine (Tue, 25 Apr 2017 14:29:39 GMT):
@yacovm I'm looking for more info on the differences between the way peers talk to each other -- "information-scoped messages" as referenced here https://gerrit.hyperledger.org/r/#/c/8273/1 by @elli-androulaki -- when the peers were given their certs by multiple CA instances as compared to entirely separate MSPs (operating in one consortium/network whathaveyou)
joe-alewine (Tue, 25 Apr 2017 14:31:17 GMT):
How would it influence decision making on behalf of potential users about whether to go with a single MSP with multiple instances as compared to multiple MSPs. I'm new to the docs team so I realize this might not be the ideal question/way to phrase the question
yacovm (Tue, 25 Apr 2017 14:33:10 GMT):
can you point me to the line please?
joe-alewine (Tue, 25 Apr 2017 14:34:54 GMT):
Lines 76 and 85. And I'm sorry, it's "organization scoped messages" not "information"
yacovm (Tue, 25 Apr 2017 14:35:18 GMT):
ah ok I'll explain briefly ok?
joe-alewine (Tue, 25 Apr 2017 14:35:24 GMT):
Sure. I appreciate it
yacovm (Tue, 25 Apr 2017 14:35:46 GMT):
So in gossip there are messages that are sent between peers that are about a specific channel - in example, ledger blocks, or metadata information
yacovm (Tue, 25 Apr 2017 14:36:10 GMT):
and the rest are messages sent between peers just for maintaining the infrastructure itself, like membership
joe-alewine (Tue, 25 Apr 2017 14:36:44 GMT):
Maintaining it how? A "hey is everyone still here?" kind of thing?
yacovm (Tue, 25 Apr 2017 14:36:48 GMT):
yeah
yacovm (Tue, 25 Apr 2017 14:36:53 GMT):
ah wait you're in the docs team
yacovm (Tue, 25 Apr 2017 14:36:57 GMT):
with Nick and co?
joe-alewine (Tue, 25 Apr 2017 14:37:03 GMT):
Yeah. He gave me the think
joe-alewine (Tue, 25 Apr 2017 14:37:07 GMT):
link
yacovm (Tue, 25 Apr 2017 14:37:12 GMT):
ah
yacovm (Tue, 25 Apr 2017 14:37:29 GMT):
so basically
yacovm (Tue, 25 Apr 2017 14:37:50 GMT):
diff. MSPs are diff orgs
rmohta (Tue, 25 Apr 2017 14:39:59 GMT):
Has joined the channel.
joe-alewine (Tue, 25 Apr 2017 14:40:26 GMT):
Yeah I've got that. I'm working on gathering some information on a "best practices" doc and I wanted some more info on the differences. Is this ability to "maintain the infrastructure" a feature important enough that it might influence network starters on how they set up membership services?
yacovm (Tue, 25 Apr 2017 14:40:45 GMT):
so
yacovm (Tue, 25 Apr 2017 14:41:18 GMT):
I'm a tad busy at the moment but since you're in the docs team we simply do a google hangout at a later point today?
yacovm (Tue, 25 Apr 2017 14:41:47 GMT):
it's easier to explain that way
joe-alewine (Tue, 25 Apr 2017 14:43:11 GMT):
I've never done a google hangout before but I'll try to figure how
joe-alewine (Tue, 25 Apr 2017 15:01:37 GMT):
@yacovm Doesn't seem too difficult, though of course saying that is tempting fate. If there's a time later today that's convenient, give me a heads up
joe-alewine (Tue, 25 Apr 2017 15:01:46 GMT):
Thanks
ashutosh_kumar (Tue, 25 Apr 2017 15:02:15 GMT):
Has joined the channel.
yacovm (Tue, 25 Apr 2017 15:04:38 GMT):
maybe later on today
yacovm (Tue, 25 Apr 2017 15:04:43 GMT):
I'm just really busy atm
joe-alewine (Tue, 25 Apr 2017 15:05:13 GMT):
I gotcha. No worries. And if not, tomorrow
joe-alewine (Tue, 25 Apr 2017 15:05:13 GMT):
I gotcha. No worries. And if not, some other time
yacovm (Tue, 25 Apr 2017 16:46:05 GMT):
@joe-alewine you still there?
joe-alewine (Tue, 25 Apr 2017 16:55:27 GMT):
Stepped out for a few minutes. Just got back
joe-alewine (Tue, 25 Apr 2017 16:55:35 GMT):
@yacovm
yacovm (Tue, 25 Apr 2017 16:55:54 GMT):
hmm ok so want to do a quick hangout?
rangak (Wed, 26 Apr 2017 04:05:44 GMT):
Has joined the channel.
raghavsood (Wed, 26 Apr 2017 04:23:20 GMT):
Has left the channel.
swangbj (Wed, 26 Apr 2017 09:35:11 GMT):
Has joined the channel.
LoveshHarchandani (Thu, 27 Apr 2017 06:44:54 GMT):
Has joined the channel.
mskim (Fri, 28 Apr 2017 07:29:39 GMT):
Has joined the channel.
wsh_bob (Fri, 28 Apr 2017 08:12:13 GMT):
Has joined the channel.
vugranam (Fri, 28 Apr 2017 12:13:30 GMT):
Has joined the channel.
jkirke (Fri, 28 Apr 2017 15:04:17 GMT):
Has left the channel.
s.narayanan (Mon, 01 May 2017 14:27:25 GMT):
Has joined the channel.
s.narayanan (Mon, 01 May 2017 14:33:27 GMT):
is it possible to turn gossip off ? Specifically we have a scenario where we have 2 orgs with 2 endorser nodes, each node being set up to receive blocks directly from orderer. In this scenario, the client is asking if gossip can be turned off altogether. Appreciate instructions on how to do this. Thanks
yacovm (Mon, 01 May 2017 14:38:34 GMT):
Can you share why is the client asking such a request?
Is there some other problem that he's trying to avoid?
yacovm (Mon, 01 May 2017 14:40:25 GMT):
Is it a logging verbosity issue?
You can make all peers connect directly to the ordering service by toggling these 2 environment variables:
https://github.com/hyperledger/fabric/blob/master/sampleconfig/core.yaml#L74-L83
The above makes the peer connect to the ordering service regardless what other peers say.
s.narayanan (Mon, 01 May 2017 14:57:58 GMT):
They are just interested in knowing if this is mandatory to use because in a small scale deployment they see it as unnecessary overhead. They understand as the number of nodes scale , there is benefit to using it. Presume that if you want peers to connect to orderer, you need to specify use leader election: false and orgleader: true. However I presume in this case gossip is still used under the covers?
yacovm (Mon, 01 May 2017 15:12:50 GMT):
It is always used if you have anchor peers
yacovm (Mon, 01 May 2017 15:12:55 GMT):
Or bootstrap peers
s.narayanan (Mon, 01 May 2017 15:41:37 GMT):
@yacovm thanks. Logical question is if it necessary to specify either (anchor or bootstrap). Is it fair to say there is no way to disable gossip?
yacovm (Mon, 01 May 2017 16:34:45 GMT):
well if you have 0 anchor peers in the channel
yacovm (Mon, 01 May 2017 16:34:50 GMT):
in every channel
yacovm (Mon, 01 May 2017 16:34:59 GMT):
and also no bootstrap peers
yacovm (Mon, 01 May 2017 16:35:09 GMT):
then gossip doesn't do anything
s.narayanan (Mon, 01 May 2017 17:27:37 GMT):
@yacovm thanks
yacovm (Mon, 01 May 2017 17:27:49 GMT):
sure
himansri (Thu, 04 May 2017 11:27:17 GMT):
Has joined the channel.
William.Z (Thu, 04 May 2017 13:05:13 GMT):
Has joined the channel.
akash42145 (Mon, 08 May 2017 04:08:14 GMT):
Has joined the channel.
amber-zhang (Tue, 09 May 2017 00:53:46 GMT):
Has joined the channel.
bhargav18 (Tue, 09 May 2017 22:39:35 GMT):
Has joined the channel.
thakkarparth007 (Wed, 10 May 2017 12:50:35 GMT):
Has joined the channel.
silentspark (Thu, 11 May 2017 08:06:20 GMT):
Has joined the channel.
rmohta (Fri, 12 May 2017 06:27:56 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=eGDdJq3P4nmY8mp9x) @troyronda @elli-androulaki So today we cannot differentiate if a transaction was initiated by a client or a peer? Or which client (if I have multiple)?
elli-androulaki (Fri, 12 May 2017 07:02:07 GMT):
@rmohta, this is correct; at least the type of the identity is not exposed directly by the interface of the MSP. However, one can leverage the OU field in X.509 certificates and an OU suffice/prefix to denote a peer/client.
rmohta (Fri, 12 May 2017 18:13:59 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=5nQvaHydFMF7LfZsA) @elli-androulaki Cool. One last question - is that (OU field) something we can extract in chaincode? Or it should be SDK's job to send that as part of invoke to chain code?
elli-androulaki (Sat, 13 May 2017 18:07:54 GMT):
@rmohta, the chaincode can obtain the transaction creator's identity (marschalled SerializedIdentity structure). SerializedIdentity consists of the msp identifier of the MSP that identity belongs to and (for the default and only MSP supported currently by Fabric) a marshalled X.509 certificate.
The OU field is exportable currently only by manually (at the chaincode side) unmarshalling the creator's identity and the subsequent X.509 cert structure and leveraging X.509 cert lib to obtain the OU field.
In the future this will be easier as we will provide either some extension to the SHIM or some invocable system chaincode that would provide membership service provider calls based on the state of each MSP on the channel/chain.
elli-androulaki (Sat, 13 May 2017 18:07:54 GMT):
@rmohta, the chaincode can obtain the transaction creator's identity (marschalled SerializedIdentity structure). SerializedIdentity consists of the msp identifier of the MSP that identity belongs to and (for the default and only MSP supported currently by Fabric) a marshalled X.509 certificate.
The OU field is exportable currently only by manually (at the chaincode side) unmarshalling the creator's identity and the subsequent X.509 cert structure and leveraging X.509 cert lib to obtain the OU field.
In the future this will be easier as we aim to provide either some extension to the SHIM or some invocable (system?) chaincode that would provide membership service provider calls based on the state of each MSP on the channel/chain. A proposal for the design of this will be out post v1.
Amjadnz (Mon, 15 May 2017 15:07:16 GMT):
Has joined the channel.
Amjadnz (Mon, 15 May 2017 15:10:28 GMT):
@elli-androulaki - thanks by the way it was very insightful.
Willson (Tue, 16 May 2017 05:33:55 GMT):
hello guys , maybe this question is a bit silly, does the orderer delivers blocks to commiter by gossip?
rocket.cat (Tue, 16 May 2017 05:33:55 GMT):
Well hello there, Willson
yacovm (Tue, 16 May 2017 06:34:12 GMT):
@Willson some peers connect to the ordering service themselves and some not.
Those that do not, receive blocks from other peers, via gossip
Willson (Tue, 16 May 2017 06:57:00 GMT):
thanks yacovm, how the peers that connect to the ordering service get block ?
yacovm (Tue, 16 May 2017 08:45:14 GMT):
@Willson via gRPC.
in short - The peer sends a `Deliver` request to the ordering service with a descriptor of what to fetch (channel, which block to start from, etc.) and then it keeps the connection open and whenever new blocks are formed, they are pushed to the peer.
Senthil1 (Tue, 16 May 2017 08:53:14 GMT):
Are only anchor peers of an organization allowed to gossip with (anchor) peers of other organizations?
Senthil1 (Tue, 16 May 2017 08:53:14 GMT):
Are only anchor peers of an organization allowed to gossip with (anchor) peers of other organizations? @yacovm
Willson (Tue, 16 May 2017 09:13:54 GMT):
@yacovm As I expected. but what port does the orderer connect to with peers? 7050? except for this not found in the configtx.yaml file and env variable
yacovm (Tue, 16 May 2017 09:23:32 GMT):
The peers connect to the ordering service, port 7050 but obviously that's configurable
yacovm (Tue, 16 May 2017 09:23:37 GMT):
it is found
yacovm (Tue, 16 May 2017 09:24:00 GMT):
https://github.com/hyperledger/fabric/blob/master/sampleconfig/configtx.yaml#L139
yacovm (Tue, 16 May 2017 09:24:03 GMT):
@Willson ^^
Willson (Tue, 16 May 2017 09:28:16 GMT):
yep, it is existing in configtx.yaml, both the sdk client and peers connecting to ordering service are used 7050?
yacovm (Tue, 16 May 2017 10:20:25 GMT):
yes
Michal Malka (Tue, 16 May 2017 11:14:15 GMT):
Has joined the channel.
Willson (Tue, 16 May 2017 11:48:07 GMT):
@yacovm thank you for your replys
Amjadnz (Thu, 18 May 2017 06:03:54 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=DCxJ25nuZRX5qR3Yi) @elli-androulaki @yacovm some help needed here - if you can please? I want to decode the x509 cert from the chaincode. Using docker environment. Once I instantiate the chaincode I am calling the stub.GetCreator() method and passing to the x509.ParseCertificate - it is complaining about not finding msp.SerializedIdentity.
Amjadnz (Thu, 18 May 2017 06:04:04 GMT):
```x509.ParseCertificate(msp.SerializedIdentity(stub.GetCreator()).IdBytes())```
Amjadnz (Thu, 18 May 2017 06:05:03 GMT):
```Error: Error endorsing chaincode: rpc error: code = 2 desc = Error starting container: Failed to generate platform-specific docker build: Error returned from build: 2 "# github.com/hyperledger/fabric/examples/chaincode/go/tts/sc/v1/userrole
chaincode/input/src/github.com/hyperledger/fabric/examples/chaincode/go/tts/sc/v1/userrole/UserRole.go:203: undefined: msp.SerializedIdentity
"```
Amjadnz (Thu, 18 May 2017 06:05:52 GMT):
When i try to force the proto directory (where msp resides in hyperledger source) in YAML file - get the following error
Amjadnz (Thu, 18 May 2017 06:06:10 GMT):
```Error: Error endorsing chaincode: rpc error: code = 2 desc = Error starting container: Failed to generate platform-specific docker build: Error returned from build: 1 "chaincode/input/src/github.com/hyperledger/fabric/protos/peer/admin.pb.go:73:8: cannot find package "github.com/golang/protobuf/proto" in any of:
/opt/go/src/github.com/golang/protobuf/proto (from $GOROOT)
/chaincode/input/src/github.com/golang/protobuf/proto (from $GOPATH)
/opt/gopath/src/github.com/golang/protobuf/proto
chaincode/input/src/github.com/hyperledger/fabric/protos/peer/admin.pb.go:76:8: cannot find package "github.com/golang/protobuf/ptypes/empty" in any of:
/opt/go/src/github.com/golang/protobuf/ptypes/empty (from $GOROOT)
/chaincode/input/src/github.com/golang/protobuf/ptypes/empty (from $GOPATH)
/opt/gopath/src/github.com/golang/protobuf/ptypes/empty
chaincode/input/src/github.com/hyperledger/fabric/protos/peer/chaincode.pb.go:10:8: cannot find package "github.com/golang/protobuf/ptypes/timestamp" in any of:
/opt/go/src/github.com/golang/protobuf/ptypes/timestamp (from $GOROOT)
/chaincode/input/src/github.com/golang/protobuf/ptypes/timestamp (from $GOPATH)
/opt/gopath/src/github.com/golang/protobuf/ptypes/timestamp
chaincode/input/src/github.com/hyperledger/fabric/protos/peer/init.go:19:8: cannot find package "github.com/op/go-logging" in any of:
/opt/go/src/github.com/op/go-logging (from $GOROOT)
/chaincode/input/src/github.com/op/go-logging (from $GOPATH)
/opt/gopath/src/github.com/op/go-logging
chaincode/input/src/github.com/hyperledger/fabric/protos/peer/admin.pb.go:80:2: cannot find package "google.golang.org/grpc" in any of:
/opt/go/src/google.golang.org/grpc (from $GOROOT)
/chaincode/input/src/google.golang.org/grpc (from $GOPATH)
/opt/gopath/src/google.golang.org/grpc
"
Usage:
```
yacovm (Thu, 18 May 2017 06:11:38 GMT):
Thats not how toy do it
yacovm (Thu, 18 May 2017 06:11:38 GMT):
Thats not how you do it
yacovm (Thu, 18 May 2017 06:20:34 GMT):
You cant give the x509 library input that is a fabric struct
yacovm (Thu, 18 May 2017 06:26:41 GMT):
The SerializedIdentity has 2 fields
yacovm (Thu, 18 May 2017 06:26:47 GMT):
you need to take the one that isn't an MSPID
yacovm (Thu, 18 May 2017 06:27:01 GMT):
and use that as raw bytes to the `ParseCertficate`
yacovm (Thu, 18 May 2017 06:27:01 GMT):
and use that as raw bytes to the `ParseCertficate`
Something like `x509.ParseCertificate(stub.GetCreator().IdBytes())`
yacovm (Thu, 18 May 2017 06:27:11 GMT):
@Amjadnz ^
bmkor (Thu, 18 May 2017 09:53:24 GMT):
Has joined the channel.
Amjadnz (Thu, 18 May 2017 09:55:03 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=c4kPpxrCvo9Wfh9MK) @yacovm Doing exactly the same. Default peer images are complaining of not able to find msp.SerializedIdentity. Do we need to change something in yaml tonpoint to hyperledger source?
Amjadnz (Thu, 18 May 2017 09:56:15 GMT):
Ok got it what you were saying
Amjadnz (Thu, 18 May 2017 09:56:22 GMT):
thanks man would try again
yacovm (Thu, 18 May 2017 09:58:12 GMT):
Sure
bmkor (Thu, 18 May 2017 10:03:06 GMT):
Hi folks. Say if I got a peer removed from the network for some days and then rejoin, what would happen on that peer? I guess it would first connect to the anchor peer and then sync the chain information from another peer. How that rejoined peer to decide which peer to get information from?
yacovm (Thu, 18 May 2017 10:42:21 GMT):
What do you mean information @bmkor ?
bmkor (Thu, 18 May 2017 10:43:30 GMT):
like the missing ledger states?
bmkor (Thu, 18 May 2017 10:43:30 GMT):
like the missing ledger data?
yacovm (Thu, 18 May 2017 10:44:23 GMT):
ah
yacovm (Thu, 18 May 2017 10:44:40 GMT):
so it will get it from some peer in the channel
bmkor (Thu, 18 May 2017 10:45:12 GMT):
the some peers are just some randomly chosen?
bmkor (Thu, 18 May 2017 10:45:12 GMT):
those peers are just some randomly chosen?
yacovm (Thu, 18 May 2017 10:47:44 GMT):
randomly chosen but from peers in the channel
bmkor (Thu, 18 May 2017 10:49:07 GMT):
Thanks. I just found this on the rst
```
Gossip-based broadcasting operates by peers receiving messages from other peers on the channel, and then forwarding these messages to a number of randomly-selected peers on the channel, where this number is a configurable constant.
```
bmkor (Thu, 18 May 2017 10:49:29 GMT):
Thanks a lot. @yacovm :)
Amjadnz (Thu, 18 May 2017 13:13:06 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=aRR4tEpov9KJyxhcN) @yacovm - can you please advise on the below?
Amjadnz (Thu, 18 May 2017 13:13:22 GMT):
the error message is
Amjadnz (Thu, 18 May 2017 13:13:25 GMT):
```Error: Error endorsing chaincode: rpc error: code = 2 desc = Error starting container: Failed to generate platform-specific docker build: Error returned from build: 1 "chaincode/input/src/github.com/hyperledger/fabric/protos/peer/admin.pb.go:73:8: cannot find package "github.com/golang/protobuf/proto" in any of:
/opt/go/src/github.com/golang/protobuf/proto (from $GOROOT)
/chaincode/input/src/github.com/golang/protobuf/proto (from $GOPATH)
/opt/gopath/src/github.com/golang/protobuf/proto
chaincode/input/src/github.com/hyperledger/fabric/protos/peer/admin.pb.go:76:8: cannot find package "github.com/golang/protobuf/ptypes/empty" in any of:
/opt/go/src/github.com/golang/protobuf/ptypes/empty (from $GOROOT)
/chaincode/input/src/github.com/golang/protobuf/ptypes/empty (from $GOPATH)
/opt/gopath/src/github.com/golang/protobuf/ptypes/empty
chaincode/input/src/github.com/hyperledger/fabric/protos/peer/chaincode.pb.go:10:8: cannot find package "github.com/golang/protobuf/ptypes/timestamp" in any of:
/opt/go/src/github.com/golang/protobuf/ptypes/timestamp (from $GOROOT)
/chaincode/input/src/github.com/golang/protobuf/ptypes/timestamp (from $GOPATH)
/opt/gopath/src/github.com/golang/protobuf/ptypes/timestamp
chaincode/input/src/github.com/hyperledger/fabric/protos/peer/init.go:19:8: cannot find package "github.com/op/go-logging" in any of:
/opt/go/src/github.com/op/go-logging (from $GOROOT)
/chaincode/input/src/github.com/op/go-logging (from $GOPATH)
/opt/gopath/src/github.com/op/go-logging
chaincode/input/src/github.com/hyperledger/fabric/protos/peer/admin.pb.go:80:2: cannot find package "google.golang.org/grpc" in any of:
/opt/go/src/google.golang.org/grpc (from $GOROOT)
/chaincode/input/src/google.golang.org/grpc (from $GOPATH)
/opt/gopath/src/google.golang.org/grpc
"
```
Amjadnz (Thu, 18 May 2017 13:14:44 GMT):
And my code is as follows:
Amjadnz (Thu, 18 May 2017 13:14:47 GMT):
```x509Cert, err := x509.ParseCertificate(stub.GetCreator().IdBytes())
if len(x509Cert.Subject.CommonName) > 0 {
fmt.Printf("Common Name of the cert :\n%s\n", x509Cert.Subject.CommonName)
}
```
yacovm (Thu, 18 May 2017 13:16:35 GMT):
Not really
yacovm (Thu, 18 May 2017 13:17:42 GMT):
Also this isn't gossip related
yacovm (Thu, 18 May 2017 13:17:50 GMT):
ask in #fabric-peer-endorser-committer
bmkor (Thu, 18 May 2017 14:14:05 GMT):
In the documentation on Gossip messaging:
bmkor (Thu, 18 May 2017 14:14:45 GMT):
"Peers maintain channel membership by collecting these alive messages; if no peer receives an alive message from a specific peer, this "dead" peer is eventually purged from channel membership."
bmkor (Thu, 18 May 2017 14:15:10 GMT):
Where are these alive or dead information saved?
C0rWin (Thu, 18 May 2017 14:17:30 GMT):
@bmkor this information is not persisted anywhere, it's maintained internally inside discovery layer
bmkor (Fri, 19 May 2017 01:22:13 GMT):
I got a further question. Say a new peer, after joining a channel, connected to an anchor peer of to a peer network (size: `N+1`), assuming the number of randomly-selected peers on the channel has been set to `n`, what would be the expected time for this new peer forming a clique? [ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=eq2FtxoYq6jftS8T7) @yacovm
bmkor (Fri, 19 May 2017 01:22:13 GMT):
I got a further question. Say a new peer, after joining a channel, connected to an anchor peer of to a peer network (size: `N+1`), assuming the number of randomly-selected peers on the channel has been set to `n`, what would be the expected time for this new peer forming a clique (N >> n)? [ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=eq2FtxoYq6jftS8T7) @yacovm
bmkor (Fri, 19 May 2017 01:22:13 GMT):
I got a further question. Say a new peer, after joining a channel, connected to an anchor peer of to a peer network (size: `N+1`), assuming the number of randomly-selected peers on the channel has been set to `n`, what would be the expected time for this new peer forming a clique (`N >> n`)? [ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=eq2FtxoYq6jftS8T7) @yacovm
bmkor (Fri, 19 May 2017 01:22:13 GMT):
I got a further question. Say a new peer, after joining a channel, connected to an anchor peer of to a peer network (size: `N+1`), assuming the number of randomly-selected peers on the channel has been set to `n` where `N >> n` , what would be the expected time for this new peer talked to all peers at least once? [ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=eq2FtxoYq6jftS8T7) @yacovm
bmkor (Fri, 19 May 2017 01:22:13 GMT):
I got a further question. Say a new peer, after joining a channel, connected to an anchor peer of to a peer network (size: `N+1`), assuming the number of randomly-selected peers on the channel has been set to `n` where `N >> n` , what would be the expected time for this new peer talked to all peers at least once (say network traffic latency and processing time are negligible) [ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=eq2FtxoYq6jftS8T7) @yacovm
bmkor (Fri, 19 May 2017 01:22:13 GMT):
I got a further question. Say a new peer, after joining a channel, connected to an anchor peer of a peer network (size: `N+1`), assuming the number of randomly-selected peers on the channel has been set to `n` where `N >> n` , what would be the expected time for this new peer talked to all peers at least once (say network traffic latency and processing time are negligible) [ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=eq2FtxoYq6jftS8T7) @yacovm
bmkor (Fri, 19 May 2017 01:22:13 GMT):
I got a further question. Say a new peer, after joining a channel, connected to an anchor peer of a peer network (size: `N+1`), assuming the number of randomly-selected peers on the channel has been set to `n` where `N >> n` , what would be the expected time for this new peer talked to all peers at least once (say network traffic latency and processing time are negligible) or if the randomness means uniform, each peer got a chance of `n/N` being selected, then the expected time for visiting all peers once would be `N*(n/N) = n`? [ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=eq2FtxoYq6jftS8T7) @yacovm
bmkor (Fri, 19 May 2017 01:22:13 GMT):
I got a further question. Say a new peer, after joining a channel, connected to an anchor peer of a peer network (size: `N+1`), assuming the number of randomly-selected peers on the channel has been set to `n` where `N >> n` , what would be the expected time for this new peer talked to all peers at least once (say network traffic latency and processing time are negligible) ? [ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=eq2FtxoYq6jftS8T7) @yacovm
bmkor (Fri, 19 May 2017 01:22:13 GMT):
I got a further question. Say a new peer, after joining a channel, connected to an anchor peer of a peer network (size: `N+1`), assuming the number of randomly-selected peers on the channel has been set to `n` where `N >> n` , what would be the expected time for this new peer talked to all peers at least once (say network traffic latency and processing time are negligible) ? My thought: suppose the randomness is uniform, the expected time for choosing a particular peer, r, would be `E(T_r) = 1*(n/N) = n/N` and the expected total time to choose all peers is `Sum E(T_r) = N*(n/N) = N` [ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=eq2FtxoYq6jftS8T7) @yacovm
bmkor (Fri, 19 May 2017 01:22:13 GMT):
I got a further question. Say a new peer, after joining a channel, connected to an anchor peer of a peer network (size: `N+1`), assuming the number of randomly-selected peers on the channel has been set to `n` where `N >> n` , what would be the expected time for this new peer talked to all peers at least once (say network traffic latency and processing time are negligible) ? [ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=eq2FtxoYq6jftS8T7) @yacovm
yacovm (Fri, 19 May 2017 05:12:30 GMT):
10 seconds
bmkor (Fri, 19 May 2017 05:14:35 GMT):
Constant? 10 sec[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=NmmJFQ7RzDFnS2ArG) @yacovm
yacovm (Fri, 19 May 2017 06:29:57 GMT):
well every node sends an alive message once per about 5 seconds
yacovm (Fri, 19 May 2017 06:30:12 GMT):
so if you're a new node in the network
yacovm (Fri, 19 May 2017 06:30:26 GMT):
5 seconds passed and you send an alive to other nodes
yacovm (Fri, 19 May 2017 06:30:55 GMT):
after 5 seconds these nodes would send you (but indirectly) an alive message too
yacovm (Fri, 19 May 2017 06:31:07 GMT):
so you "know about them" after that
yacovm (Fri, 19 May 2017 06:31:22 GMT):
if you asked how much until you have actually communicated with all of them
yacovm (Fri, 19 May 2017 06:32:13 GMT):
depends if you want an expectence (E(X)) or an upper bound. But why do you ask? Why is this important?
bmkor (Fri, 19 May 2017 09:18:53 GMT):
Thanks @yacovm I meant to know how long will a peer expect to wait if in a peer network, one of the peers got the most updated ledger data.
bmkor (Fri, 19 May 2017 09:18:53 GMT):
Thanks @yacovm I meant to estimate how long will a peer to wait if in a peer network, one of the peers got the most updated ledger data.
bmkor (Fri, 19 May 2017 09:18:53 GMT):
Thanks @yacovm I meant to estimate how long will a peer have to wait if in a peer network, one of the peers just got updated its ledger data.
bmkor (Fri, 19 May 2017 09:25:46 GMT):
Ah I misunderstood. The update process is from orderer broadcasting. My bad. Please ignore my question.
yacovm (Fri, 19 May 2017 09:28:39 GMT):
No it is also from peers
yacovm (Fri, 19 May 2017 09:28:53 GMT):
Am not at home at the momemt, hard to elaborate
anik (Fri, 19 May 2017 09:49:49 GMT):
Has joined the channel.
yacovm (Fri, 19 May 2017 09:56:15 GMT):
@bmkor so basically a peer may get blocks from the ordering service directly or via some other peer
bmkor (Fri, 19 May 2017 09:57:48 GMT):
you meant those peer committers?
yacovm (Fri, 19 May 2017 10:16:56 GMT):
Every peer is a committer
bmkor (Fri, 19 May 2017 10:33:44 GMT):
Let me try to understand this from reading the codes.
Amjadnz (Fri, 19 May 2017 11:09:15 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=WptF8Ha2DbomtLdP2) @Amjadnz Resolved it by parsing the cert into pem encoding and all went fine.
C0rWin (Fri, 19 May 2017 15:25:34 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=bsxmvYrWPCzazxnLZ) @bmkor expected time of peer to talk with others is of order O(logN).
bmkor (Fri, 19 May 2017 23:42:09 GMT):
Thanks
Amjadnz (Sat, 20 May 2017 08:48:36 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=u87kDJC24vWvA2RPp) @yacovm Thanks - Following up wit CC-DEV channel
Jay (Mon, 22 May 2017 10:26:24 GMT):
Has joined the channel.
bmalavan (Wed, 24 May 2017 08:12:48 GMT):
Has joined the channel.
srvnnp (Wed, 24 May 2017 13:39:20 GMT):
Has joined the channel.
duwenhui (Fri, 26 May 2017 06:09:12 GMT):
Has joined the channel.
guruce (Mon, 05 Jun 2017 07:29:04 GMT):
Has joined the channel.
AdnanC (Mon, 05 Jun 2017 19:49:22 GMT):
After a leading peer has been elected, how do I know which peer became leader?
yacovm (Mon, 05 Jun 2017 19:50:32 GMT):
read the logs?
yacovm (Mon, 05 Jun 2017 19:50:46 GMT):
read the logs
AdnanC (Mon, 05 Jun 2017 19:51:24 GMT):
ok thanks
AdnanC (Mon, 05 Jun 2017 19:51:50 GMT):
I believe there is no other direct way?
yacovm (Mon, 05 Jun 2017 19:53:06 GMT):
not without writing lots of code
AdnanC (Mon, 05 Jun 2017 19:53:24 GMT):
ok
jdockter (Mon, 05 Jun 2017 20:52:33 GMT):
Has left the channel.
smithbk (Wed, 07 Jun 2017 15:46:41 GMT):
Has left the channel.
DDmitry (Thu, 08 Jun 2017 14:24:21 GMT):
Has joined the channel.
AdnanC (Thu, 08 Jun 2017 20:02:06 GMT):
@yacovm https://jira.hyperledger.org/browse/FAB-4475
Calvin_Heo (Fri, 09 Jun 2017 09:01:33 GMT):
Has joined the channel.
AdnanC (Fri, 09 Jun 2017 17:57:01 GMT):
What is the "default" state of gossip leader and gossip election? in other words, if someone downloads Fabric, builds the docker images, and ups the network without anything on the dockercompose files regarding gossip-----what will be the value of leder_election and is_leader?
AdnanC (Fri, 09 Jun 2017 17:57:01 GMT):
What is the "default" state of gossip leader and gossip election? in other words, if someone downloads Fabric, builds the docker images, and ups the network without anything on the dockercompose files regarding gossip-----what will be the value of leader_election and is_leader? @yacovm
AdnanC (Fri, 09 Jun 2017 17:57:01 GMT):
What is the "default" state of gossip leader and gossip election? in other words, if someone downloads Fabric, builds the docker images, and ups the network without anything on the dockercompose files regarding gossip-----what will be the value of leder_election and is_leader? @yacovm
AdnanC (Fri, 09 Jun 2017 17:58:02 GMT):
is it going to use `sampleconfig/core.yaml` ands use the values there?
yacovm (Fri, 09 Jun 2017 18:14:47 GMT):
yeah
jrezwan (Sun, 11 Jun 2017 16:16:54 GMT):
Has joined the channel.
william123 (Mon, 12 Jun 2017 13:44:41 GMT):
Has joined the channel.
AdnanC (Mon, 12 Jun 2017 21:35:54 GMT):
@yacovm https://jira.hyperledger.org/browse/FAB-4586 this is very similar to the other one I filed last week.
yacovm (Mon, 12 Jun 2017 21:45:50 GMT):
@AdnanC so when you queried for the value in step (4) you did it for the peer that was restarted?
AdnanC (Mon, 12 Jun 2017 21:46:09 GMT):
yes
AdnanC (Mon, 12 Jun 2017 21:47:28 GMT):
1 sec, let me check if I attached the correct one, (I did both)
AdnanC (Mon, 12 Jun 2017 21:47:28 GMT):
1 sec, let me check if I attached the correct one, (I did both leader and non leader)
AdnanC (Mon, 12 Jun 2017 21:49:07 GMT):
yes, the non leader, the one that was restarted
yacovm (Mon, 12 Jun 2017 21:50:23 GMT):
so you stopped the non leader in org2, did an invoke on org1, then started the peer in org2 that was restarted and waited for the query to give the right result
yacovm (Mon, 12 Jun 2017 21:50:24 GMT):
right?
yacovm (Mon, 12 Jun 2017 22:12:06 GMT):
@AdnanC I tried doing exactly what you did but can't reproduce :(
yacovm (Mon, 12 Jun 2017 22:27:13 GMT):
Message Attachments
yacovm (Mon, 12 Jun 2017 22:27:28 GMT):
that's all I did and it works for me... verified twice
yacovm (Mon, 12 Jun 2017 22:28:48 GMT):
Message Attachments
lm_nop (Mon, 12 Jun 2017 23:14:32 GMT):
Has joined the channel.
lprao (Wed, 14 Jun 2017 04:09:37 GMT):
Has joined the channel.
lprao (Wed, 14 Jun 2017 04:09:48 GMT):
Hi,
I am using fabric(1.0.0-alpha) in "dev" mode on a classic TwoOrg topology with 4 peers, 2 CAs, 1 SDK(fabric-sdk-go). When i initiate a transaction(chaincode_example02) to peer1(in Org1), peer1 is able to execute the chaincode but peer2(also in Org1) sees a "No such channel" error -
[36m2017-06-06 00:58:01.312 UTC [gossip/gossip#23.0.0.12:7056] handleMessage -> DEBU 228[0m Entering, [156 43 33 171 114 249 57 151 235 35 197 207 85 206 60 90 178 209 11 90 101 229 107 30 108 214 245 107 90 240 230 193] sent us GossipMessage: channel:"testchannel" tag:CHAN_OR_ORG state_info_pull_req:<> , Envelope: 18 bytes, Signature: 0 bytes
[36m2017-06-06 00:58:01.313 UTC [gossip/gossip#23.0.0.12:7056] handleMessage -> DEBU 229[0m No such channel [116 101 115 116 99 104 97 110 110 101 108] discarding message GossipMessage: channel:"testchannel" tag:CHAN_OR_ORG state_info_pull_req:<> , Envelope: 18 bytes, Signature: 0 bytes
[36m2017-06-06 00:58:01.313 UTC [gossip/gossip#23.0.0.12:7056] handleMessage -> DEBU 22a[0m Exiting
[36m2017-06-06 00:58:01.320 UTC [gossip/gossip#23.0.0.12:7056] handleMessage -> DEBU 22b[0m Entering, [156 43 33 171 114 249 57 151 235 35 197 207 85 206 60 90 178 209 11 90 101 229 107 30 108 214 245 107 90 240 230 193] sent us GossipMessage: channel:"testchannel" tag:CHAN_OR_ORG state_info:
yacovm (Wed, 14 Jun 2017 06:17:38 GMT):
This is a debug messgae
yacovm (Wed, 14 Jun 2017 06:17:41 GMT):
Message
yacovm (Wed, 14 Jun 2017 06:17:56 GMT):
It doesnt mean you should fix this
yacovm (Wed, 14 Jun 2017 06:18:23 GMT):
Also, @lprao what do you mean by dev mode?
lprao (Wed, 14 Jun 2017 08:30:51 GMT):
@yacovm By "dev" mode i mean all peers are running as linux processes(--peer-chaincodedev==true) and not in docker containers. The actual command is - "./build/bin/peer node start --peer-chaincodedev=true --peer-defaultchain=false >> /tmp/fabric_peer_console.log 2>&1 &"
yacovm (Wed, 14 Jun 2017 08:31:08 GMT):
ah so first of all
yacovm (Wed, 14 Jun 2017 08:31:11 GMT):
the default chain thing
yacovm (Wed, 14 Jun 2017 08:31:16 GMT):
it doesn't exist anymore
yacovm (Wed, 14 Jun 2017 08:31:22 GMT):
and second- ok now I get what you mean
yacovm (Wed, 14 Jun 2017 08:31:28 GMT):
yeah there is indeed that dev mode thing
yacovm (Wed, 14 Jun 2017 08:31:41 GMT):
anyway I wouldn't worry about it - it's DEBUG message
yacovm (Wed, 14 Jun 2017 08:31:42 GMT):
not WARN
lprao (Wed, 14 Jun 2017 08:34:38 GMT):
hmm, but peer2 does not execute the chaincode(chaincode_example02) and so the ledger is empty
yacovm (Wed, 14 Jun 2017 08:34:48 GMT):
it has nothing to do with gossip
lprao (Wed, 14 Jun 2017 08:35:05 GMT):
ok
lprao (Wed, 14 Jun 2017 08:36:06 GMT):
which is the right channel to discuss this issue? i have tried chaincode-dev but haven't heard back.
yacovm (Wed, 14 Jun 2017 08:36:34 GMT):
what issue
lprao (Wed, 14 Jun 2017 08:37:08 GMT):
the chaincode not running on peer2.
yacovm (Wed, 14 Jun 2017 08:37:26 GMT):
ah I guess either #fabric or #chaincode-dev
lprao (Wed, 14 Jun 2017 08:37:43 GMT):
ok, thanks.
s.narayanan (Fri, 16 Jun 2017 14:52:41 GMT):
How do I disable gossip protocol? Specifically scenario where the peers need to connect directly to the orderer to deliver block. Appreciate instruction/guidance to do this
yacovm (Fri, 16 Jun 2017 14:53:32 GMT):
you can take these 2 configuration keys https://github.com/hyperledger/fabric/blob/master/sampleconfig/core.yaml#L87-L92
yacovm (Fri, 16 Jun 2017 14:53:40 GMT):
set `orgLeader` to true
yacovm (Fri, 16 Jun 2017 14:53:45 GMT):
and the `useLeaderElection` to false
yacovm (Fri, 16 Jun 2017 14:54:03 GMT):
but this doesn't really disable gossip, just makes all the peers to connect to the ordering service
s.narayanan (Fri, 16 Jun 2017 14:56:05 GMT):
@yacovm thanks. since we are using docker images for peers presume this has to be done through an environment variable setting in docker compose file?
yacovm (Fri, 16 Jun 2017 14:56:26 GMT):
yes.
yacovm (Fri, 16 Jun 2017 14:56:31 GMT):
make all the characters uppercase
yacovm (Fri, 16 Jun 2017 14:56:34 GMT):
and also
yacovm (Fri, 16 Jun 2017 14:56:44 GMT):
prefix with: CORE_PEER_GOSSIP
yacovm (Fri, 16 Jun 2017 14:56:55 GMT):
i.e CORE_PEER_GOSSIP_ORGLEADER=true
s.narayanan (Fri, 16 Jun 2017 15:00:47 GMT):
@yacovm thanks. Presume it would also be CORE_PEER_GOSSIP_USELEADERELECTION=false. Also what is the reason disabling gossip protocol is not supported? If all peers are leaders one can presumably avoid the chattiness of gossip protocol?
yacovm (Fri, 16 Jun 2017 15:01:21 GMT):
just don't put anything in the anchor peers
yacovm (Fri, 16 Jun 2017 15:01:26 GMT):
and don't put anything in the bootstrap peers
yacovm (Fri, 16 Jun 2017 15:01:30 GMT):
and this effectively disables
silliman (Fri, 16 Jun 2017 15:07:29 GMT):
@yacovm what do you mean by 'don't put anything in the anchor peers' do you mean don't actually define any, or something else? (same question for bootstrap peers)
yacovm (Fri, 16 Jun 2017 15:09:35 GMT):
exactly
silliman (Fri, 16 Jun 2017 15:10:08 GMT):
not to define any
yacovm (Fri, 16 Jun 2017 15:10:24 GMT):
yes
sfukazu (Mon, 19 Jun 2017 06:34:53 GMT):
Has joined the channel.
roj (Mon, 19 Jun 2017 10:42:35 GMT):
Has joined the channel.
reubent 1 (Mon, 19 Jun 2017 16:15:07 GMT):
Has joined the channel.
reubent 1 (Mon, 19 Jun 2017 16:17:25 GMT):
Hello! I'm getting a stack dump when the gossip service is not liking the certificate chain. I'm obviously doing something silly but I am pretty sure this still shouldn't happen:
```
2017-06-19 16:14:20.954 UTC [sccapi] RegisterSysCC -> INFO 02a system chaincode qscc(github.com/hyperledger/fabric/core/chaincode/qscc) registered
2017-06-19 16:14:20.954 UTC [nodeCmd] serve -> DEBU 02b Running peer
2017-06-19 16:14:20.954 UTC [msp] GetLocalMSP -> DEBU 02c Returning existing local MSP
2017-06-19 16:14:20.954 UTC [msp] GetLocalMSP -> DEBU 02d Returning existing local MSP
2017-06-19 16:14:20.954 UTC [msp] GetDefaultSigningIdentity -> DEBU 02e Obtaining default signing identity
2017-06-19 16:14:20.955 UTC [gossip/service] func1 -> INFO 02f Initialize gossip with endpoint peer0 and bootstrap set [127.0.0.1:7051]
2017-06-19 16:14:20.955 UTC [msp] GetLocalMSP -> DEBU 030 Returning existing local MSP
2017-06-19 16:14:20.955 UTC [msp] DeserializeIdentity -> INFO 031 Obtaining identity
2017-06-19 16:14:20.955 UTC [msp/identity] newIdentity -> DEBU 032 Creating identity instance for ID &{GospelTest c05fd73ae4c3fa9bf21f53780b33796d7e9888790f24e3b92ac0d3c0b6096c2a}
2017-06-19 16:14:20.955 UTC [msp] GetLocalMSP -> DEBU 033 Returning existing local MSP
2017-06-19 16:14:20.955 UTC [msp] Validate -> DEBU 034 MSP GospelTest validating identity
2017-06-19 16:14:20.956 UTC [msp] getCertificationChain -> DEBU 035 MSP GospelTest getting certification chain
panic: runtime error: index out of range
goroutine 1 [running]:
panic(0xc9c680, 0xc4200160a0)
/opt/go/src/runtime/panic.go:500 +0x1a1
github.com/hyperledger/fabric/gossip/integration.newConfig(0xc42020a688, 0x5, 0xc42000c442, 0xa, 0xc42033d770, 0x1, 0x1, 0x20)
/opt/gopath/src/github.com/hyperledger/fabric/gossip/integration/integration.go:37 +0x6ea
github.com/hyperledger/fabric/gossip/integration.NewGossipComponent(0xc420084b00, 0x52b, 0x580, 0xc42020a688, 0x5, 0xc42016e780, 0x13d3fa0, 0xc42033d940, 0x13e45e0, 0xc420330d50, ...)
/opt/gopath/src/github.com/hyperledger/fabric/gossip/integration/integration.go:79 +0x93
github.com/hyperledger/fabric/gossip/service.InitGossipServiceCustomDeliveryFactory.func1()
/opt/gopath/src/github.com/hyperledger/fabric/gossip/service/gossip_service.go:146 +0x41d
sync.(*Once).Do(0x1423388, 0xc4201536b8)
/opt/go/src/sync/once.go:44 +0xdb
github.com/hyperledger/fabric/gossip/service.InitGossipServiceCustomDeliveryFactory(0xc420084b00, 0x52b, 0x580, 0xc42020a688, 0x5, 0xc42016e780, 0x13d3ea0, 0x14232b8, 0x13e45e0, 0xc420330d50, ...)
/opt/gopath/src/github.com/hyperledger/fabric/gossip/service/gossip_service.go:157 +0x139
github.com/hyperledger/fabric/gossip/service.InitGossipService(0xc420084b00, 0x52b, 0x580, 0xc42026d050, 0xf, 0xc42016e780, 0x13e45e0, 0xc420330d50, 0x13d3fa0, 0xc42033d940, ...)
/opt/gopath/src/github.com/hyperledger/fabric/gossip/service/gossip_service.go:129 +0x13a
github.com/hyperledger/fabric/peer/node.serve(0x14232b8, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hyperledger/fabric/peer/node/start.go:174 +0x71c
github.com/hyperledger/fabric/peer/node.glob..func1(0x13ca260, 0x14232b8, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hyperledger/fabric/peer/node/start.go:70 +0x3f
github.com/hyperledger/fabric/vendor/github.com/spf13/cobra.(*Command).execute(0x13ca260, 0x14232b8, 0x0, 0x0, 0x13ca260, 0x14232b8)
/opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/spf13/cobra/command.go:599 +0x234
github.com/hyperledger/fabric/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x13ca8c0, 0xf, 0xc420010455, 0xa)
/opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/spf13/cobra/command.go:689 +0x367
github.com/hyperledger/fabric/vendor/github.com/spf13/cobra.(*Command).Execute(0x13ca8c0, 0x1b, 0xc420010455)
/opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/spf13/cobra/command.go:648 +0x2b
main.main()
/opt/gopath/src/github.com/hyperledger/fabric/peer/main.go:117 +0x54e
```
reubent 1 (Mon, 19 Jun 2017 16:17:57 GMT):
Any ideas?
yacovm (Mon, 19 Jun 2017 16:21:54 GMT):
uh, can you please open a JIRA and upload the `sampleconfig` folder
yacovm (Mon, 19 Jun 2017 16:22:03 GMT):
and tag me (@ yacovm) in the JIRA?
reubent 1 (Mon, 19 Jun 2017 16:23:30 GMT):
Hi, Yacov, more than happy to do so - I'm not running using the sample config though
reubent 1 (Mon, 19 Jun 2017 16:23:40 GMT):
I can give you a copy of the MSP if it helps?
reubent 1 (Mon, 19 Jun 2017 16:34:51 GMT):
https://jira.hyperledger.org/browse/FAB-4867
Thank you :)
yacovm (Mon, 19 Jun 2017 16:35:03 GMT):
that's what I meant @reubent 1
yacovm (Mon, 19 Jun 2017 16:44:35 GMT):
@C0rWin ^
C0rWin (Mon, 19 Jun 2017 21:26:54 GMT):
@reubent 1 in `core.yaml` you have attached into FAB-4867
C0rWin (Mon, 19 Jun 2017 21:27:00 GMT):
``` # Gossip related configuration
gossip:
# Bootstrap set to initialize gossip with.
# This is a list of peers that the peer reaches out to at startup.
# Important: The endpoints here have to be endpoints of peers in the same
# organization, because the peer would refuse connecting to these endpoints
# unless they are in the same organization as the peer.
bootstrap: 127.0.0.1:7051
# NOTE: orgLeader and useLeaderElection parameters are mutual exclusive
# setting both to true would result in the termination of the peer, since this is undefined
# state.
# Defines whenever peer will initialize dynamic algorithm for
# "leader" selection, where leader is the peer to establish
# connection with ordering service and use delivery protocol
# to pull ledger blocks from ordering service
useLeaderElection: false
# Statically defines peer to be an organization "leader",
# where this means that current peer will maintain connection
# with ordering service and disseminate block across peers in
# its own organization
orgLeader: true
# ID of this instance
endpoint: peer0
# Maximum count of blocks we store in memory```
C0rWin (Mon, 19 Jun 2017 21:27:15 GMT):
the endpoint parameter missing port number
C0rWin (Mon, 19 Jun 2017 21:27:21 GMT):
hence the failure
C0rWin (Mon, 19 Jun 2017 21:28:08 GMT):
I will add a patch to provide more meaningful error message and handle the failure more graceful, rather than panicking with simple stack trace
bh4rtp (Tue, 20 Jun 2017 05:45:37 GMT):
Has joined the channel.
bh4rtp (Tue, 20 Jun 2017 05:57:21 GMT):
hi, i have a question. the e2e_cli example network is used. i want to deploy peer0, pee1, orderer, couchdb on manager host, and peer2, peer3 and cli on worker host.
to simplify the deploy, the volume option is skipped by below steps:
1. start up network_setup.sh and do not run any command, i.e. orderer, peer node start.
2. backup volume directories using `tar cvf fabric.tar ./dirs` on the containers and export the containers into tar files.
3. import the tar files, run the images and restore volume directories by `tar xvf fabric.tar`
4. export again the containers, now the images will have all the files required to run.
the docker swarm are installed on the two hosts. and a overlay network is created on the manager host. when starting the network, and `docker logs peer0`, there is a panic error.
bh4rtp (Tue, 20 Jun 2017 05:57:21 GMT):
hi, i have a question. the e2e_cli example network is used. i want to deploy peer0, pee1, orderer, couchdb on manager host, and peer2, peer3 and cli on worker host.
to simplify the deploy, the volume option is skipped by below steps:
1. start up network_setup.sh and do not run any command, i.e. orderer, peer node start.
2. backup volume directories using `tar cvf fabric.tar ./dirs` on the containers and export the containers into tar files.
3. import the tar files, run the images and restore volume directories by `tar xvf fabric.tar`
4. export again the containers, now the images will have all the files required to run.
the docker swarm is installed on the two hosts. and an overlay network named `hyp-net` is created on the manager host. when starting the network, and `docker logs peer0`, there is a panic error.
bh4rtp (Tue, 20 Jun 2017 05:57:42 GMT):
```2017-06-19 19:42:40.160 CST [gossip/discovery] handleAliveMessage -> ERRO 1ce Bad configuration detected: Received AliveMessage from a peer with the same PKI-ID as myself: tag:EMPTY alive_msg:
bh4rtp (Tue, 20 Jun 2017 05:58:36 GMT):
what is the cause of this problem?
reubent 1 (Tue, 20 Jun 2017 08:18:22 GMT):
@C0rWin Many thanks - I was sure it was something I was doing wrong
dongqi (Tue, 20 Jun 2017 08:35:36 GMT):
Has joined the channel.
rcyrus (Wed, 21 Jun 2017 19:13:25 GMT):
Has joined the channel.
SDChoi (Fri, 23 Jun 2017 02:21:13 GMT):
Has joined the channel.
scottz (Fri, 23 Jun 2017 15:52:32 GMT):
@yacovm I think it was you who previously provided to me some helpful information. I have 2 followup questions:
scottz (Fri, 23 Jun 2017 15:52:54 GMT):
"A reasoning behind having bootstrap peers is to peers of the organization to find each other and
to know each other before you make some of them join the channel."
---> QUESTION: Why is this helpful?
"Another reasoning is that technically you can create a channel without any anchor peer in your organization.
So having a bootstrap peer(s) for your org is very very important in such a case."
---> QUESTION: Why would you do this, in production network?
yacovm (Fri, 23 Jun 2017 15:53:03 GMT):
I try not to provide unhelpful information if I can
yacovm (Fri, 23 Jun 2017 15:55:06 GMT):
> Why is this helpful?
Because if you take a look, for example - in the e2e - first you create a channel without any anchor peers, and then you send 2 subsequent config transactions that add anchor peers.
Now, in case the peers are configured to use leader election, without bootstrap peers they won't see each other at channel join, and will all connect to the ordering service, until the config update that adds anchor peers would happen for each organization.
yacovm (Fri, 23 Jun 2017 15:55:36 GMT):
Now if they had anchor peers, they could upon channel join - do leader election, and then not all to connect to the ordering service
yacovm (Fri, 23 Jun 2017 15:56:31 GMT):
> QUESTION: Why would you do this, in production network?
because you have access control over the config items of various organizations, so the admin that creates the channel, say from orgA - can't add into the genesis block, anchor peer of orgB
yacovm (Fri, 23 Jun 2017 16:00:15 GMT):
Also another important thing
yacovm (Fri, 23 Jun 2017 16:00:25 GMT):
if you have a set of peers, and they are configured not to use leader election
yacovm (Fri, 23 Jun 2017 16:00:35 GMT):
and only 1 of them is configured to be a leader with `orgLeader=true`
yacovm (Fri, 23 Jun 2017 16:00:38 GMT):
the rest is false
yacovm (Fri, 23 Jun 2017 16:00:46 GMT):
well, and you don't have bootstrap peers
yacovm (Fri, 23 Jun 2017 16:01:15 GMT):
the ones that have the `orgLeader=false` won't be able to get the config block that adds anchor peers
scottz (Fri, 23 Jun 2017 16:02:52 GMT):
In the second question - ok, so it makes sense that each admin could add their own anchor peer. But, if all the peers in one particular organization get restarted at same time, or a new org is added, then I think they would need to know an anchor peer in ANOTHER org, or else they couldn't sync up state database. Hmmm... or maybe could they simply get all the necessary blocks from the orderer again, and (re)create their state database that way?
yacovm (Fri, 23 Jun 2017 16:04:36 GMT):
an anchor peer in another org works though
yacovm (Fri, 23 Jun 2017 16:04:39 GMT):
what's the problem?
yacovm (Fri, 23 Jun 2017 16:05:01 GMT):
when peers are restarted they don't need to re-create their state database
yacovm (Fri, 23 Jun 2017 16:05:03 GMT):
it's persisted
yacovm (Fri, 23 Jun 2017 16:05:20 GMT):
unless you take out the disks or wipe out the file system
scottz (Fri, 23 Jun 2017 16:08:50 GMT):
Just trying to think of various (not-so-happy-path) scenarios. So... it seems we really DO need to use anchor peers, one in each org, or we could not reconfig to add a new org to an existing network.
yacovm (Fri, 23 Jun 2017 16:09:38 GMT):
no, that's not true
yacovm (Fri, 23 Jun 2017 16:09:43 GMT):
you can always use leaderElection
yacovm (Fri, 23 Jun 2017 16:09:48 GMT):
and then you don't need anchor peers
yacovm (Fri, 23 Jun 2017 16:09:48 GMT):
and then you don't need all anchor peers
yacovm (Fri, 23 Jun 2017 16:10:02 GMT):
then, they are just "nice to have"
scottz (Fri, 23 Jun 2017 16:16:40 GMT):
Using bootstrap peers and leader election would allow the peers in an org to work together, but they could not sync state database from other orgs without contacting an anchor peer in another org. Wouldn't they need to do that? it seems you are saying, in the case of restarting all peers in one org, that they would not need to do so. But in the case of adding a new org, is it necessary? (or maybe not because the new org should NOT have access to the blocks that existed before the new org was added.)
scottz (Fri, 23 Jun 2017 16:16:40 GMT):
Using bootstrap peers and leader election would allow the peers in an org to work together, but they could not sync state database from other orgs without contacting an anchor peer in another org. Wouldn't they need to do that? it seems you are saying, in the case of restarting all peers in one org, that they would not need to do so.
yacovm (Fri, 23 Jun 2017 16:27:40 GMT):
well but don't forget - you just need 1 anchor peer
yacovm (Fri, 23 Jun 2017 16:27:42 GMT):
to be alive
yacovm (Fri, 23 Jun 2017 16:27:55 GMT):
and if all peers know about it
yacovm (Fri, 23 Jun 2017 16:27:59 GMT):
they can contact it
yacovm (Fri, 23 Jun 2017 16:28:12 GMT):
an anchor peer isn't a gateway or something
scottz (Fri, 23 Jun 2017 17:38:51 GMT):
OK. But if there are no anchor peers anywhere, and if we are booting the first new peer in a channel in a newly added org, could that peer get all blocks from orderer? In other words, can we function without anchorpeers, and should we test that configuration? (I have heard that if the ledger is huge, then the orderer might not deliver all the blocks in the channel ledger, which means the peer would be required to sync the whole blockchain from another peer, which means an anchor peer in another organziation, since this is the first peer to come up in a new org.)
yacovm (Fri, 23 Jun 2017 17:42:07 GMT):
well if there are no anchor peers anywhere and you're booting the first new peer in a channel on a new org, so obviously that peer doesn't know anyone else but itself, right?
yacovm (Fri, 23 Jun 2017 17:42:35 GMT):
so it can only obtain blocks from the ordering service
yacovm (Fri, 23 Jun 2017 17:42:41 GMT):
but I don't see where you're getting at
yacovm (Fri, 23 Jun 2017 17:43:42 GMT):
> which means the peer would be required to sync the whole blockchain from another peer, which means an anchor peer in another organziation, since this is the first peer to come up in a new org
Well it doesn't work that way. You not always choose to sync from an anchor peer. It depends on if you know other peers or not
yacovm (Fri, 23 Jun 2017 17:44:38 GMT):
also you do it in "batches"
yacovm (Fri, 23 Jun 2017 17:44:44 GMT):
each batch you select a peer again
yacovm (Fri, 23 Jun 2017 17:44:56 GMT):
so you probably won't select the same peer over and over
yacovm (Fri, 23 Jun 2017 17:45:01 GMT):
unless it's.... the only one you know
clessor (Fri, 23 Jun 2017 19:23:27 GMT):
Has joined the channel.
scottz (Fri, 23 Jun 2017 21:25:53 GMT):
@yacovm ok, that helps. In summary: If we want to configure a network, without anchor peers, it could work. In that case, we would have to use a bootstrap peer in each org, and every other initialized peer in an org would contact the bootstrap_peer. The first orgleader peer in an org would retrieve the whole blockchain from the orderer, and new peers would sync state from that one (or from neighbor peers).
scottz (Fri, 23 Jun 2017 22:18:35 GMT):
Someone else asks: in my scenario, where we start and join first peer in a newly added org (and it is orgleader), when the other orgs do have anchor peers: the peer is the only one in its org. Does it sync from anchor peer(s) in other orgs, or does it read/deliver all blocks from orderer, or both?
C0rWin (Fri, 23 Jun 2017 22:41:16 GMT):
leader election is per channel and organization, hence if peer is the only one node in the org, it will be declared a leader, connect to the ordering service and pull the blocks
yacovm (Sat, 24 Jun 2017 06:40:32 GMT):
Message Attachments
yacovm (Sat, 24 Jun 2017 06:40:42 GMT):
@scottz I edited my message above
yacovm (Sat, 24 Jun 2017 06:40:55 GMT):
In case that wasn't clear, I wasn't implying you don't need APs if you have LE on
scottz (Mon, 26 Jun 2017 16:01:43 GMT):
New questions. We are considering a test (without leader election) to set exactly 2 leaders per org, instead of just one. Is that recommended? I would think so, in case one stops. And the peers would get their blocks from whichever leader is operating, right?
scottz (Mon, 26 Jun 2017 16:01:43 GMT):
New questions. We are considering a test (without leader election) to set exactly 2 leaders per org, instead of just one. Is that recommended? I would think so, in case one of the orgleader peers stops. And the nonleader peers in an org would get their blocks from whichever leader is operating, right?
yacovm (Mon, 26 Jun 2017 16:02:15 GMT):
what is the scenario exactly?
yacovm (Mon, 26 Jun 2017 16:02:23 GMT):
have 2 leaders statically configured
yacovm (Mon, 26 Jun 2017 16:02:27 GMT):
and some number of other peers
yacovm (Mon, 26 Jun 2017 16:02:34 GMT):
and "stop" on of the leaders?
yacovm (Mon, 26 Jun 2017 16:02:44 GMT):
and to ensure blocks still reach the other peers?
scottz (Mon, 26 Jun 2017 16:03:04 GMT):
yes
yacovm (Mon, 26 Jun 2017 16:03:45 GMT):
You can test it, if you want
scottz (Mon, 26 Jun 2017 16:05:09 GMT):
oh, that raises another question. Do all the nonleader peers get the new blocks from an orgleader directly? Or if we had 10 peers, could some remote peers get from neighboring nonleaders.
yacovm (Mon, 26 Jun 2017 16:05:31 GMT):
the latter
scottz (Mon, 26 Jun 2017 16:08:40 GMT):
ok cool. And I guess my first question was more to get confirmation that the network of peers within an org is somewhat dynamic. e.g. if one peer gets its blocks from orgleader1, and orgleader1 goes down, then it would get blocks via orgleader2 (instead of waiting for orgleader1 to restart and retrieve all the missed blocks itself from teh orderer and forward them). I guess you answered that already that the peers could get blocks via any orgleader.
yacovm (Mon, 26 Jun 2017 16:10:47 GMT):
I was talking about multiple leaders in the same org
yacovm (Mon, 26 Jun 2017 16:10:50 GMT):
just to be sure
yacovm (Mon, 26 Jun 2017 16:10:58 GMT):
obtaining blocks from other orgs is *sloooow*
scottz (Mon, 26 Jun 2017 16:11:38 GMT):
sorry. yes that is all I meant - from other orgleaders within the same org.
scottz (Mon, 26 Jun 2017 16:12:51 GMT):
But that raises another question. I thought I saw a "fix" merged a few weeks ago that prevents blocks to be shared between orgs. I must have misunderstood, since you are saying we definitely CAN get blocks from different orgs.
yacovm (Mon, 26 Jun 2017 16:13:28 GMT):
yeah but we have 2 ways of doing that
yacovm (Mon, 26 Jun 2017 16:13:40 GMT):
it disabled one of them ;)
yacovm (Mon, 26 Jun 2017 16:14:01 GMT):
Nice tracking though
scottz (Mon, 26 Jun 2017 16:14:05 GMT):
ok.
yacovm (Mon, 26 Jun 2017 16:14:26 GMT):
if I ever want to hire a private eye I would know where to look for
scottz (Mon, 26 Jun 2017 16:14:49 GMT):
thanks. Then... are there any criteria / restrictions about the sharing across orgs?
yacovm (Mon, 26 Jun 2017 16:15:07 GMT):
yeah, the block needs to be committed into the ledger
yacovm (Mon, 26 Jun 2017 16:15:13 GMT):
only such blocks can be shared across orgs
scottz (Mon, 26 Jun 2017 16:17:22 GMT):
The only usecase I can think of ... if useLeaderElection=false and if there are no orgleaders running in one org, then those peers could get blocks from other org.
yacovm (Mon, 26 Jun 2017 16:18:14 GMT):
Well they can always get them regardless
scottz (Mon, 26 Jun 2017 16:36:43 GMT):
Based on values in fabric/sampleconfig/core.yaml (except all peers have useLeaderElection=true and orgLeader=false), when I stop an orgleader peer, how long can I expect for the next leader to be elected (worst case)? Is it membershipSampleInterval + leaderAliveThreshold + leaderElectionDuration (16s)?
SotirisAlfonsos (Tue, 27 Jun 2017 10:58:23 GMT):
Has joined the channel.
SotirisAlfonsos (Tue, 27 Jun 2017 11:00:04 GMT):
Hello. I want to change the default gossip port for peers:7051 to make it org specific. Could someone help me with that?
SotirisAlfonsos (Tue, 27 Jun 2017 11:00:04 GMT):
Hello. I want to change the default gossip port for peers:7051 to make it org specific. Could someone help me with that?
```peer0.org3.example.com | 2017-06-27 11:46:02.597 UTC [gossip/discovery] func1 -> WARN 1a0 Could not connect to {127.0.0.1:7051 [] [] 127.0.0.1:7051} : x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs
peer1.org2.example.com | 2017-06-27 11:46:03.982 UTC [gossip/discovery] func1 -> WARN 19f Could not connect to {127.0.0.1:7051 [] [] 127.0.0.1:7051} : x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs
peer1.org3.example.com | 2017-06-27 11:46:04.743 UTC [gossip/discovery] func1 -> WARN 1a0 Could not connect to {127.0.0.1:7051 [] [] 127.0.0.1:7051} : x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs
peer0.org2.example.com | 2017-06-27 11:46:06.277 UTC [gossip/discovery] func1 -> WARN 19f Could not connect to {127.0.0.1:7051 [] [] 127.0.0.1:7051} : x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs
```
yacovm (Tue, 27 Jun 2017 12:43:10 GMT):
Do you mean
yacovm (Tue, 27 Jun 2017 12:43:13 GMT):
only for gossip protocol
yacovm (Tue, 27 Jun 2017 12:43:27 GMT):
or for all the peer gRPC services (proposals from clients, etc.) ?
SotirisAlfonsos (Tue, 27 Jun 2017 12:49:10 GMT):
@yacovm the idea is to redirect certain connections, so all peer grpc services. I think client to docker to docker is easy. I think my error is coming from not setting up the gossip properly. For example, in my case, i do not know why org2 tries to connect to the port specified for org1.
SotirisAlfonsos (Tue, 27 Jun 2017 12:49:10 GMT):
@yacovm the idea is to redirect certain connections, so all peer grpc services. I think client to docker is easy. I think my error is coming from not setting up the gossip properly. For example, in my case, i do not know why org2 tries to connect to the port specified for org1.
SotirisAlfonsos (Tue, 27 Jun 2017 12:49:10 GMT):
@yacovm the idea is to redirect certain connections, so all peer grpc services. I think client to docker is easy. My error is coming from not setting up the gossip properly. For example, in my case, i do not know why org2 tries to connect to the port specified for org1.
yacovm (Tue, 27 Jun 2017 13:06:42 GMT):
Did you try to set CORE_PEER_ENDPOINT?
SotirisAlfonsos (Tue, 27 Jun 2017 13:11:21 GMT):
@yacovm Yes but the problem seems to come from CORE_PEER_ADDRESS. Let me try again though
SotirisAlfonsos (Tue, 27 Jun 2017 13:56:39 GMT):
No does not work.
webdaford (Tue, 27 Jun 2017 18:49:06 GMT):
Has joined the channel.
rezamt (Thu, 29 Jun 2017 04:22:56 GMT):
Has joined the channel.
SotirisAlfonsos (Thu, 29 Jun 2017 09:57:08 GMT):
Hello. I am getting a connect failed when i try to connect to a peer with a port other than `7051`
My peer settings.
```peer0.org2.example.com:
container_name: peer0.org2.example.com
extends:
file: base.yaml
service: peer-base
environment:
- CORE_PEER_ID=peer0.org2.example.com
- CORE_PEER_LOCALMSPID=Org2MSP
- CORE_PEER_ADDRESS=peer0.org2.example.com:8051
- CORE_PEER_GOSSIP_ENDPOINT=peer0.org2.example.com:8051
- CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer0.org2.example.com:8051
- CORE_PEER_GOSSIP_BOOTSTRAP=peer0.org2.example.com:8051
ports:
- 8051:8051
- 8053:8053
volumes:
- ./crypto-config/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/:/etc/hyperledger/crypto/peer
depends_on:
- orderer.example.com```
SotirisAlfonsos (Thu, 29 Jun 2017 09:57:08 GMT):
Hello. I am getting a connect failed when i try to connect to a peer with a port other than `7051`
My peer settings.
```peer0.org2.example.com:
container_name: peer0.org2.example.com
extends:
file: base.yaml
service: peer-base
environment:
- CORE_PEER_ID=peer0.org2.example.com
- CORE_PEER_LOCALMSPID=Org2MSP
- CORE_PEER_ADDRESS=peer0.org2.example.com:8051
- CORE_PEER_GOSSIP_ENDPOINT=peer0.org2.example.com:8051
- CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer0.org2.example.com:8051
- CORE_PEER_GOSSIP_BOOTSTRAP=peer0.org2.example.com:8051
ports:
- 8051:8051
- 8053:8053
volumes:
- ./crypto-config/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/:/etc/hyperledger/crypto/peer
depends_on:
- orderer.example.com```
Any suggestions?
yacovm (Thu, 29 Jun 2017 10:11:29 GMT):
Are you telling me that it still binds to 7051?
yacovm (Thu, 29 Jun 2017 10:12:03 GMT):
ahhhh
yacovm (Thu, 29 Jun 2017 10:12:07 GMT):
peer.listenAddress
yacovm (Thu, 29 Jun 2017 10:12:11 GMT):
@SotirisAlfonsos
yacovm (Thu, 29 Jun 2017 10:12:25 GMT):
change the CORE_PEER_LISTENADDRESS
SotirisAlfonsos (Thu, 29 Jun 2017 10:42:15 GMT):
@yacovm yep that was it. Now the peers are getting the message but the response takes way to long. I will have to play with it a bit more. Thanks for the help
yacovm (Thu, 29 Jun 2017 10:42:26 GMT):
sure
scottz (Fri, 30 Jun 2017 19:15:54 GMT):
How does an orgLeader peer choose which orderer to connect to? is it random? is there a peer or gossip log that indicates which orderer address/port?
scottz (Fri, 30 Jun 2017 19:15:54 GMT):
How does an orgLeader peer choose which orderer to connect to? is it random choice of one of the orderers in the genesis block? is there a peer or gossip log that indicates which orderer address/port?
C0rWin (Fri, 30 Jun 2017 19:27:03 GMT):
@scottz yes, the choice of orderer endpoint is random selection
ariannagolf (Fri, 30 Jun 2017 20:36:58 GMT):
Has joined the channel.
alburt (Mon, 03 Jul 2017 10:49:16 GMT):
Has joined the channel.
SotirisAlfonsos (Tue, 04 Jul 2017 08:06:11 GMT):
Hello, i would like to block the communication from the orderer to one of my peers? Any suggestions on how can i approach that?
yacovm (Tue, 04 Jul 2017 08:19:29 GMT):
what do you mean exactly block the communication
yacovm (Tue, 04 Jul 2017 08:19:32 GMT):
and why?
yacovm (Tue, 04 Jul 2017 08:19:49 GMT):
you simply want the peer not to connect to the ordering service or you want it not to be able to connect at all?
SotirisAlfonsos (Tue, 04 Jul 2017 08:23:33 GMT):
@yacovm I want to test my network with network delays and package loss. The only way i see, apart from blocking the entire veth of a peer, is to block a specific communication channel.
What i am trying to do is make the orderer broadcast eventually not reach one peer (e.g. peer0).
yacovm (Tue, 04 Jul 2017 08:23:53 GMT):
@C0rWin
yacovm (Tue, 04 Jul 2017 08:23:58 GMT):
^
yacovm (Tue, 04 Jul 2017 08:24:53 GMT):
what about iptable rules?
yacovm (Tue, 04 Jul 2017 08:24:55 GMT):
it'll be the simplest
C0rWin (Tue, 04 Jul 2017 08:25:15 GMT):
@SotirisAlfonsos consider to take a look on this https://github.com/worstcase/blockade
C0rWin (Tue, 04 Jul 2017 08:25:47 GMT):
you can have decent network setup with docker, while leveraging blockade you can test different network disruption use cases
C0rWin (Tue, 04 Jul 2017 08:26:03 GMT):
duplicate packets, slow networking, segmentation
SotirisAlfonsos (Tue, 04 Jul 2017 08:29:48 GMT):
I am trying to use iptables but it does not seem to have the effect that i want on the network. @C0rWin i was not aware of this tool, seems like exactly what i need. Will take a look at it today. Thank you both
SotirisAlfonsos (Tue, 04 Jul 2017 11:43:35 GMT):
@C0rWin nice tool but as far as i have tested i can isolate a container like with tc or iptables on the veth of a container. But i do not want to delay every incoming connection. I would like to delay only the path from a -> b, and not everything that goes to b. But thank you anyway. I am open for new suggestions.
qizhang (Wed, 05 Jul 2017 15:04:52 GMT):
Has joined the channel.
qizhang (Wed, 05 Jul 2017 15:06:41 GMT):
@yacovm Hi, I have a question regarding to whether and how a peer can detect a missing block in its ledger. In Fabric, each peer maintains a full copy of the ledger/blockchain (let's assume there is only one channel in the network). What will happen if a peer lost one of its blocks in the blockchain (e.g. another process on the peer accidentally wiped out the disk area that stores the block)? Will this peer be able to detect that this block is missing? If yes, how will the peer detect it and how is it going to repair? Thanks!
yacovm (Wed, 05 Jul 2017 15:07:31 GMT):
as long as you have the first block and you're able to communicate with peers
yacovm (Wed, 05 Jul 2017 15:07:38 GMT):
you will reach some peer
yacovm (Wed, 05 Jul 2017 15:07:44 GMT):
and sync from its blocks
qizhang (Wed, 05 Jul 2017 15:08:24 GMT):
@yacovm I see, but how is a peer going to detect that a block is missing? Does a peer periodically scan its ledger?
qizhang (Wed, 05 Jul 2017 15:08:24 GMT):
I see, but how is a peer going to detect that a block is missing? Is the peer going to periodically scan its ledger?
qizhang (Wed, 05 Jul 2017 15:08:24 GMT):
@yacovm I see, but how is a peer going to detect that a block is missing? Is the peer going to periodically scan its ledger?
yacovm (Wed, 05 Jul 2017 15:10:13 GMT):
it doesn't need to periodically scan
yacovm (Wed, 05 Jul 2017 15:10:19 GMT):
it keeps track of the block number
yacovm (Wed, 05 Jul 2017 15:10:24 GMT):
and sees the block numbers of his friends
yacovm (Wed, 05 Jul 2017 15:10:28 GMT):
they gossip this data
qizhang (Wed, 05 Jul 2017 15:14:55 GMT):
@yacovm Thanks! I understand, but still not that clear. Let's say in a network of 4 peers (peer1 to peer4), peer1 - peer3 all have block 1,2,3,4,5,6, but peer 4 only has block 1,3,4,5,6. When will peer 4 see the block numbers of the other peers and find that it is missing block 2?
qizhang (Wed, 05 Jul 2017 15:14:55 GMT):
@yacovm Thanks! I understand more, but still not quite clear. Let's say in a network of 4 peers (peer1 to peer4), peer1 - peer3 all have block 1,2,3,4,5,6, but peer 4 only has block 1,3,4,5,6. When will peer 4 see the block numbers of the other peers and find that it is missing block 2?
qizhang (Wed, 05 Jul 2017 15:14:55 GMT):
@yacovm Thanks! I understand, but still not quite clear. Let's say in a network of 4 peers (peer1 to peer4), peer1 - peer3 all have block 1,2,3,4,5,6, but peer 4 only has block 1,3,4,5,6. When will peer 4 see the block numbers of the other peers and find that it is missing block 2?
yacovm (Wed, 05 Jul 2017 15:22:51 GMT):
when?
yacovm (Wed, 05 Jul 2017 15:23:13 GMT):
well it will see they have block 4 after a few seconds
divyank (Wed, 05 Jul 2017 19:05:21 GMT):
How can I enable gossip across peers from different orgs?
yacovm (Wed, 05 Jul 2017 20:18:32 GMT):
define the external endpoint
yacovm (Wed, 05 Jul 2017 20:18:36 GMT):
in the config
divyank (Wed, 05 Jul 2017 20:57:14 GMT):
@yacovm Do I also need to configure TLS settings to ensure an Org1 peer can talk to an Org2 peer? (I'm using a default network setup like fabric/examples/e2e_cli)
yacovm (Wed, 05 Jul 2017 20:57:35 GMT):
good question
yacovm (Wed, 05 Jul 2017 20:57:44 GMT):
when you make the peers join the channel
yacovm (Wed, 05 Jul 2017 20:57:49 GMT):
fabric will do that for you ;)
yacovm (Wed, 05 Jul 2017 20:57:58 GMT):
(magic)
divyank (Thu, 06 Jul 2017 17:10:45 GMT):
@yacovm trying to gain some insight into the magic :)
Are the trusted roots for TLS configured based on the Organization CAs for each channel?
yacovm (Thu, 06 Jul 2017 18:56:22 GMT):
Yeah
yacovm (Thu, 06 Jul 2017 18:56:33 GMT):
So basically when you define a channel
yacovm (Thu, 06 Jul 2017 18:56:44 GMT):
You put in the genesis block the root CA certs
yacovm (Thu, 06 Jul 2017 18:57:15 GMT):
When the peers get them, they update their root ca cert pool
yacovm (Thu, 06 Jul 2017 18:57:32 GMT):
For the connections between peers
DannyWong (Sat, 08 Jul 2017 03:27:35 GMT):
Has joined the channel.
eliranbi (Mon, 10 Jul 2017 17:38:14 GMT):
Has joined the channel.
eliranbi (Mon, 10 Jul 2017 17:40:47 GMT):
https://chat.hyperledger.org/channel/fabric-consensus?msg=6ftKc3fXW6fcdRJth
yacovm (Mon, 10 Jul 2017 19:09:10 GMT):
what do you mean how?
yacovm (Mon, 10 Jul 2017 19:09:15 GMT):
it is done via the same mechanism
yacovm (Mon, 10 Jul 2017 19:09:23 GMT):
the time difference doesn't matter
gauthampamu (Tue, 11 Jul 2017 21:18:02 GMT):
Has joined the channel.
scottz (Tue, 11 Jul 2017 21:54:42 GMT):
@yacov Regarding https://gerrit.hyperledger.org/r/#/c/11377/ , is there an error log that announces that the peer dropped a gossip message? It would be good to announce a block was dropped.
scottz (Tue, 11 Jul 2017 21:54:42 GMT):
@yacovm Regarding https://gerrit.hyperledger.org/r/#/c/11377/ , is there an error log that announces that the peer dropped a gossip message? It would be good to announce a block was dropped.
scottz (Tue, 11 Jul 2017 21:56:27 GMT):
Since the other peers don't know what they never received, what happens later, after dropping blocks 500-1000, when traffic slows down? Does block number 1001 get delivered and disseminated to all the peers? Do all the nonleader peers ever get the missed blocks? Won't they become confused when they notice the gap in blocknumbers?
yacovm (Tue, 11 Jul 2017 22:01:39 GMT):
It is logged
yacovm (Tue, 11 Jul 2017 22:02:08 GMT):
But that change set isn't related
yacovm (Tue, 11 Jul 2017 22:02:08 GMT):
But that change set isn't related to dropping a message
yacovm (Tue, 11 Jul 2017 22:02:46 GMT):
I think you have a mistake in the change set, Scott
scottz (Tue, 11 Jul 2017 22:05:59 GMT):
I know you didn't add a log with that changeset. I just wanted to know if there was a log - what we can look for while testing to see if a block/message gets dropped. Can you share exactly what it looks like?
yacovm (Tue, 11 Jul 2017 22:08:15 GMT):
wait
yacovm (Tue, 11 Jul 2017 22:08:20 GMT):
let's make sure we are in sync
yacovm (Tue, 11 Jul 2017 22:08:51 GMT):
are you talking about dropping messages *because of overflow (too many messages per second)* or dropping messages of blocks that are not verified?
yacovm (Tue, 11 Jul 2017 22:09:04 GMT):
the change set you linked refers to the latter
scottz (Tue, 11 Jul 2017 22:10:53 GMT):
I did mean the first one. But now that you mention it, can you answer both?
yacovm (Tue, 11 Jul 2017 22:11:25 GMT):
so the 2nd one - is printed https://gerrit.hyperledger.org/r/#/c/11377/4/gossip/gossip/channel/channel.go@428
scottz (Tue, 11 Jul 2017 22:12:18 GMT):
ok
yacovm (Tue, 11 Jul 2017 22:12:26 GMT):
the latter - nope, not printed.
https://github.com/hyperledger/fabric/blob/master/gossip/comm/conn.go#L262-L263
We can add a print post v1 in DEBUG
yacovm (Tue, 11 Jul 2017 22:12:26 GMT):
the former - nope, not printed.
https://github.com/hyperledger/fabric/blob/master/gossip/comm/conn.go#L262-L263
We can add a print post v1 in DEBUG
yacovm (Tue, 11 Jul 2017 22:13:02 GMT):
That's odd, I was sure we had a DEBUG print, I guess it got deleted
scottz (Tue, 11 Jul 2017 22:14:29 GMT):
we can create a jira for that. hmmm. If I lose a block on the send buffer, then another peer will not receive a block. That seems more serious than a DEBUG log.
yacovm (Tue, 11 Jul 2017 22:14:45 GMT):
It's not that simple
scottz (Tue, 11 Jul 2017 22:14:46 GMT):
you just compromised the ledger
yacovm (Tue, 11 Jul 2017 22:14:48 GMT):
it's much more complicated
yacovm (Tue, 11 Jul 2017 22:14:50 GMT):
nah
yacovm (Tue, 11 Jul 2017 22:15:01 GMT):
I'll elaborate
yacovm (Tue, 11 Jul 2017 22:15:48 GMT):
If you lose a block via a connection buffer overflow, the block may still get to that peer via a different peer
yacovm (Tue, 11 Jul 2017 22:16:12 GMT):
also - in gossip we have reconciliation
yacovm (Tue, 11 Jul 2017 22:16:30 GMT):
eventually you will see that you're missing a block that some other peer has
yacovm (Tue, 11 Jul 2017 22:16:38 GMT):
and you would fetch the block from that peer
yacovm (Tue, 11 Jul 2017 22:17:58 GMT):
BTW dropping messages is probably the *only thing* you can do if you have to "keep the system running".
Otherwise - the system may deadlock
yacovm (Tue, 11 Jul 2017 22:17:58 GMT):
BTW dropping messages is probably the *only thing* you can do if you have to "keep the system running".
Otherwise - the system may get "Stuck"
yacovm (Tue, 11 Jul 2017 22:18:54 GMT):
think about it - if you have a network where you input data into it in a rate that exceeds the network consumption - you will get stuck.
scottz (Tue, 11 Jul 2017 22:23:36 GMT):
hmmm. ok. then the way to test this would be to run bursty traffic for awhile, and then after a longer while can query and confirm all blocks eventually were delivered to all peers.
yacovm (Tue, 11 Jul 2017 22:24:33 GMT):
yeah
scottz (Tue, 11 Jul 2017 22:24:48 GMT):
ok. thanks. gotta run. have a good night!
SotirisAlfonsos (Fri, 14 Jul 2017 09:27:05 GMT):
Hello. Where can i define the peers that the orderer should message on broadcast? Is it `AnchorPeers`?
yifanxiang (Fri, 14 Jul 2017 09:27:38 GMT):
Has joined the channel.
C0rWin (Fri, 14 Jul 2017 09:30:37 GMT):
There is leader election mechanism which engages selection of peers which connects to ordering service, basically no need to manually define it
C0rWin (Fri, 14 Jul 2017 09:31:52 GMT):
if you need it, you can turn leader election off and make static selection, but please note, that this approach is not resilient to failures and if your peer which connects to ordering service will fails, you will have to bring him up again manually
C0rWin (Fri, 14 Jul 2017 09:32:39 GMT):
in case of leader election on, such peer will be reselected once failure will be detected (a few seconds)
C0rWin (Fri, 14 Jul 2017 09:33:10 GMT):
@SotirisAlfonsos ^^^
C0rWin (Fri, 14 Jul 2017 09:34:19 GMT):
this is a part of `core.yaml` you might be interesting in
C0rWin (Fri, 14 Jul 2017 09:34:24 GMT):
```
# Gossip related configuration
gossip:
# Bootstrap set to initialize gossip with.
# This is a list of other peers that this peer reaches out to at startup.
# Important: The endpoints here have to be endpoints of peers in the same
# organization, because the peer would refuse connecting to these endpoints
# unless they are in the same organization as the peer.
bootstrap: 127.0.0.1:7051
# NOTE: orgLeader and useLeaderElection parameters are mutual exclusive.
# Setting both to true would result in the termination of the peer
# since this is undefined state. If the peers are configured with
# useLeaderElection=false, make sure there is at least 1 peer in the
# organization that its orgLeader is set to true.
# Defines whenever peer will initialize dynamic algorithm for
# "leader" selection, where leader is the peer to establish
# connection with ordering service and use delivery protocol
# to pull ledger blocks from ordering service. It is recommended to
# use leader election for large networks of peers.
useLeaderElection: false
# Statically defines peer to be an organization "leader",
# where this means that current peer will maintain connection
# with ordering service and disseminate block across peers in
# its own organization
orgLeader: true
```
C0rWin (Fri, 14 Jul 2017 09:35:14 GMT):
you either use leader election, hence making `useLeaderElection:true` or select peer as a leader statically `orgLeader: true`....
C0rWin (Fri, 14 Jul 2017 09:35:27 GMT):
these parameters are mutual exclusive
SotirisAlfonsos (Fri, 14 Jul 2017 09:35:44 GMT):
@C0rWin hmm. Currently i am dropping packets from the orderer to peer0. But peer1 seems to not get the packets either. Leader election is on. My bootstrap is different though.
C0rWin (Fri, 14 Jul 2017 09:36:38 GMT):
> dropping packets
do you mean deliver packets?
C0rWin (Fri, 14 Jul 2017 09:36:55 GMT):
are peer0 and peer1 in same org?
C0rWin (Fri, 14 Jul 2017 09:37:18 GMT):
have you configured either of them to bootstrap set?
SotirisAlfonsos (Fri, 14 Jul 2017 09:38:26 GMT):
Yes they are in the same org. Applying package loss to the connection so they do not receive the new blocks.
SotirisAlfonsos (Fri, 14 Jul 2017 09:38:26 GMT):
Yes they are in the same org. Applying package loss to the connection so they do not receive the new blocks. Basically he can not communicate with the orderer. but peer1 experiences the same package loss. For some reason
C0rWin (Fri, 14 Jul 2017 09:39:20 GMT):
ok, but peer0 is alive and peer1 can see it?
SotirisAlfonsos (Fri, 14 Jul 2017 09:39:48 GMT):
Yes they are both alive
C0rWin (Fri, 14 Jul 2017 09:40:44 GMT):
do you know which one of them is a leader?
yacovm (Fri, 14 Jul 2017 09:49:28 GMT):
Did you join both peers to the channel?
SotirisAlfonsos (Fri, 14 Jul 2017 10:07:49 GMT):
@yacovm Yes. so let me elaborate a bit because the details i provided so far are not sufficient. I create the containers where org1 belongs to two channels and controls 2 peers ( `peer0.org1.example.com` `peer1.org1.example.com` ). I am triggering the network delays. So an iptables rule that classifies the packets going from the `orderer` to `peer0.org1.example.com`. Then with a `tc qdisc` i apply package loss 100% to those packets.
Now i run my code that goes: user enrollment, channel creation, join channels, install chaincode. Everything works up to that point. Now i am sending two instantiates, one for each channel, and sending them both to `org1`. The instantiates are signed by both `peer0.org1.example.com` `peer1.org1.example.com`, and now i would expect for the orderer to send the block to the peers. `peer0.org1.example.com` should not receive the block, and he does not. But i would expect that `peer1.org1.example.com` would receive the block, but he does not.
SotirisAlfonsos (Fri, 14 Jul 2017 10:07:49 GMT):
@yacovm Yes. so let me elaborate a bit because the details i provided so far are not sufficient. I create the containers where org1 belongs to two channels and controls 2 peers ( `peer0.org1.example.com` `peer1.org1.example.com` ). I am triggering the network delays. So an iptables rule that classifies the packets going from the `orderer` to `peer0.org1.example.com`. Then with a `tc qdisc` i apply package loss 100% to those packets.
Now i run my code that goes: user enrollment, channel creation, join channels, install chaincode. Everything works up to that point. Now i am sending two instantiates, one for each channel, and sending them both to `org1`. The instantiates are signed by both `peer0.org1.example.com` `peer1.org1.example.com`, and now i would expect for the orderer to send the block to the peers. `peer0.org1.example.com` should not receive the block, and he does not. But i would expect that `peer1.org1.example.com` would receive the block, but he does not.
I can see on the logs
```peer0.org1.example.com | 2017-07-14 09:49:19.638 UTC [ConnProducer] NewConnection -> ERRO 057 Failed connecting to orderer.example.com:7050 , error: context deadline exceeded
peer0.org1.example.com | 2017-07-14 09:49:19.639 UTC [deliveryClient] connect -> ERRO 058 Failed obtaining connection: Could not connect to any of the endpoints: [orderer.example.com:7050]```
but nothing for peer1.
SotirisAlfonsos (Fri, 14 Jul 2017 10:07:49 GMT):
@yacovm Yes. so let me elaborate a bit because the details i provided so far are not sufficient. I create the containers where org1 belongs to two channels and controls 2 peers ( `peer0.org1.example.com` `peer1.org1.example.com` ). I am triggering the network delays. So an iptables rule that classifies the packets going from the `orderer` to `peer0.org1.example.com`. Then with a `tc qdisc` i apply package loss 100% to those packets.
Now i run my code that goes: user enrollment, channel creation, join channels, install chaincode. Everything works up to that point. Now i am sending two instantiates, one for each channel, and sending them both to `org1`. The instantiates are signed by both `peer0.org1.example.com` `peer1.org1.example.com`, and now i would expect for the orderer to send the block to the peers. `peer0.org1.example.com` should not receive the block, and he does not. But i would expect that `peer1.org1.example.com` would receive the block, but he does not.
I can see on the logs
```peer0.org1.example.com | 2017-07-14 09:49:19.638 UTC [ConnProducer] NewConnection -> ERRO 057 Failed connecting to orderer.example.com:7050 , error: context deadline exceeded
peer0.org1.example.com | 2017-07-14 09:49:19.639 UTC [deliveryClient] connect -> ERRO 058 Failed obtaining connection: Could not connect to any of the endpoints: [orderer.example.com:7050]```
but nothing for peer1. As an AnchorPeer for `org1` i only have `peer0.org1.example.com`.
yacovm (Fri, 14 Jul 2017 10:32:18 GMT):
Upload the dockrfiles
yacovm (Fri, 14 Jul 2017 10:32:27 GMT):
The compose
SotirisAlfonsos (Fri, 14 Jul 2017 10:41:18 GMT):
its a bit of a mess but:
https://github.com/SotirisAlfonsos/TempKafkaSet/blob/master/docker-compose.yaml
https://github.com/SotirisAlfonsos/TempKafkaSet/blob/master/base.yaml
yacovm (Fri, 14 Jul 2017 12:16:13 GMT):
@SotirisAlfonsos let's try something shall wel?
yacovm (Fri, 14 Jul 2017 12:16:15 GMT):
*shall we
yacovm (Fri, 14 Jul 2017 12:16:24 GMT):
kill the peer that can't connect to the orderer
yacovm (Fri, 14 Jul 2017 12:16:38 GMT):
and see if the other peer - gets these drops of blocks
yacovm (Fri, 14 Jul 2017 12:16:38 GMT):
and see if the other peer - gets these droppings of packets
SotirisAlfonsos (Fri, 14 Jul 2017 12:16:58 GMT):
sounds good. give me a sec
SotirisAlfonsos (Fri, 14 Jul 2017 12:23:35 GMT):
Side note. I dropped the packets from the `orderer` to `peer1.org1.example.com`. Now i see 2 messages on peer1 failing to connect to the orderer.
```2017-07-14 12:16:13.561 UTC [ConnProducer] NewConnection -> ERRO 03b Failed connecting to orderer.example.com:7050 , error: context deadline exceeded
2017-07-14 12:16:13.561 UTC [deliveryClient] connect -> ERRO 03c Failed obtaining connection: Could not connect to any of the endpoints: [orderer.example.com:7050]```
But after that he got the blocks. I guess from `peer0.org1.example.com`
yacovm (Fri, 14 Jul 2017 12:24:14 GMT):
I don't get it
yacovm (Fri, 14 Jul 2017 12:25:23 GMT):
I told you to kill peer0
yacovm (Fri, 14 Jul 2017 12:25:32 GMT):
so how does peer1 get the blocks if peer0 is now dead?
SotirisAlfonsos (Fri, 14 Jul 2017 12:26:16 GMT):
This is a case where the only connection with package loss is `orderer` -> `peer1`. I did not try your suggestion yet. was running another test
SotirisAlfonsos (Fri, 14 Jul 2017 12:26:28 GMT):
setting it up now
yacovm (Fri, 14 Jul 2017 12:26:34 GMT):
ah
yacovm (Fri, 14 Jul 2017 12:26:48 GMT):
so it seems like @C0rWin 's theory is correct
yacovm (Fri, 14 Jul 2017 12:26:54 GMT):
try what i said though
yacovm (Fri, 14 Jul 2017 12:26:59 GMT):
start everything from scratch
yacovm (Fri, 14 Jul 2017 12:27:06 GMT):
make the packet loss from peer0 to the orderer
yacovm (Fri, 14 Jul 2017 12:27:11 GMT):
ensure peer1 doesn't get the blocks
yacovm (Fri, 14 Jul 2017 12:27:15 GMT):
and then kill peer0
yacovm (Fri, 14 Jul 2017 12:27:19 GMT):
and see that peer1 gets the blocks
yacovm (Fri, 14 Jul 2017 12:28:14 GMT):
I'm on mobile from now on so will be a bit non responsive, but pleas write your results here
SotirisAlfonsos (Fri, 14 Jul 2017 12:36:36 GMT):
Yes @C0rWin was right. After killing `peer0`, `peer1` tries to connect to him ( but fails ) and then gets the blocks, i guess because he became the leader.
SotirisAlfonsos (Fri, 14 Jul 2017 12:39:55 GMT):
But is there a way to solve that? Can i specify a timer for leader election?
yacovm (Fri, 14 Jul 2017 12:40:49 GMT):
We will
yacovm (Fri, 14 Jul 2017 12:41:07 GMT):
Can you open a jira?
SotirisAlfonsos (Fri, 14 Jul 2017 12:43:33 GMT):
I do not have an account. I will request one and create a jira
yacovm (Fri, 14 Jul 2017 12:56:32 GMT):
Ah its ok if you dont have i'll open a jira
SotirisAlfonsos (Fri, 14 Jul 2017 13:08:20 GMT):
no i did i am just searching to tag you
SotirisAlfonsos (Fri, 14 Jul 2017 13:09:17 GMT):
Great. thanks for the help
yacovm (Fri, 14 Jul 2017 13:09:18 GMT):
Same username
toddinpal (Sat, 15 Jul 2017 13:50:47 GMT):
Has joined the channel.
cre8bidio (Mon, 17 Jul 2017 00:45:53 GMT):
Has joined the channel.
qsmen (Mon, 17 Jul 2017 06:37:44 GMT):
Has joined the channel.
qsmen (Mon, 17 Jul 2017 07:08:17 GMT):
it is always said blockchain is based on p2p communication. one peer sends information to another peeer or an peer broadcast information to every peer?
dushyantbehl (Mon, 17 Jul 2017 07:57:08 GMT):
Has joined the channel.
narayanprusty (Tue, 18 Jul 2017 11:49:16 GMT):
Has joined the channel.
narayanprusty (Tue, 18 Jul 2017 11:49:21 GMT):
Does peers sync only from peers of the same organization and orderer it is connected to or do they also sync from peers of other organizations of the same channel.
Basically I want to know if peers of different organizations connect to each other and sync up.
yacovm (Tue, 18 Jul 2017 11:50:05 GMT):
all orgs of the same channel
narayanprusty (Tue, 18 Jul 2017 11:52:32 GMT):
Ok. It means yes.
yacovm (Tue, 18 Jul 2017 11:54:38 GMT):
it does
BhavishaDawda (Wed, 19 Jul 2017 19:24:06 GMT):
Has joined the channel.
linyuadam (Thu, 20 Jul 2017 05:50:07 GMT):
Has joined the channel.
rush_mwright (Thu, 20 Jul 2017 14:49:25 GMT):
Has joined the channel.
indirajith (Fri, 21 Jul 2017 10:21:16 GMT):
Has joined the channel.
dave.enyeart (Tue, 25 Jul 2017 21:00:31 GMT):
@yacovm @C0rWin Does state transfer recompute block hashes and therefore verify chain integrity as it is being transferred? If not, do you agree it would be a good feature to verify chain integrity as it goes? This would ensure that one corrupted/tampered peer would not bleed into others.
dave.enyeart (Tue, 25 Jul 2017 21:02:25 GMT):
@binhn Note that code was written to Verify the integrity of the chain for https://jira.hyperledger.org/browse/FAB-609. I'd still like us to consider it for a next release, even if we don't have cross-peer verification yet.
yacovm (Tue, 25 Jul 2017 21:02:46 GMT):
the integrity is preserved by each step
yacovm (Tue, 25 Jul 2017 21:03:06 GMT):
you can't do forward hash chain verification
yacovm (Tue, 25 Jul 2017 21:03:08 GMT):
only backwards
yacovm (Tue, 25 Jul 2017 21:03:17 GMT):
since the ledger is built onwards
yacovm (Tue, 25 Jul 2017 21:03:23 GMT):
it is verified via signatures
C0rWin (Tue, 25 Jul 2017 21:03:26 GMT):
state transfer verifies signatures of the blocks
C0rWin (Tue, 25 Jul 2017 21:03:50 GMT):
``` if err := s.mcs.VerifyBlock(common2.ChainID(s.chainID), payload.SeqNum, payload.Data); err != nil {
logger.Warningf("Error verifying block with sequence number %d, due to %s", payload.SeqNum, err)
return uint64(0), err
}```
C0rWin (Tue, 25 Jul 2017 21:04:15 GMT):
doesn't ledger do chain validation while commits the block?
C0rWin (Tue, 25 Jul 2017 21:04:15 GMT):
@dave.enyeart doesn't ledger do chain validation while commits the block?
dave.enyeart (Tue, 25 Jul 2017 21:06:44 GMT):
i mean if somebody altered data in peerA block 100 post-commit, would that tampered data flow to downstream state transferred peers?
dave.enyeart (Tue, 25 Jul 2017 21:07:56 GMT):
i meant a verification of block hashes. i guess a complete verification would require checking block hashes and signatures?
C0rWin (Tue, 25 Jul 2017 21:08:07 GMT):
how you can alter it? is signed
C0rWin (Tue, 25 Jul 2017 21:08:07 GMT):
how you can alter it? it signed
dave.enyeart (Tue, 25 Jul 2017 21:08:39 GMT):
whose signature is it? an orderer? and where is it? in metadata?
C0rWin (Tue, 25 Jul 2017 21:12:11 GMT):
orderer, and yes metadata.
C0rWin (Tue, 25 Jul 2017 21:14:01 GMT):
but again, I do not understand, didn't ledger ensures to verify hash of previous block while committing the new block?
dave.enyeart (Tue, 25 Jul 2017 21:14:53 GMT):
'ledger' only does state validation. 'committer' does other validations.
yacovm (Tue, 25 Jul 2017 21:25:45 GMT):
you mean MVCC validation @dave.enyeart
yacovm (Tue, 25 Jul 2017 21:25:51 GMT):
right?
dave.enyeart (Tue, 25 Jul 2017 22:23:13 GMT):
i mean, nothing in ledger component verifies hashes across blocks. not MVCC or any other ledger code. Discussed with @yacovm that this is acceptable since peer trusts orderer and the data is verified against orderer signature.
dave.enyeart (Tue, 25 Jul 2017 22:23:13 GMT):
i mean, nothing in ledger component verifies hashes across blocks. not MVCC or any other ledger code. Discussed with @yacovm that this is acceptable since peer trusts orderer and the block is verified against orderer signature.
dave.enyeart (Tue, 25 Jul 2017 22:25:07 GMT):
and to answer my original question... i agree that since state transfer is verifying block against orderer signature, that there is no chance to corrupted/tampered data bleeding into other peers via state transfer
dave.enyeart (Tue, 25 Jul 2017 22:25:07 GMT):
and to answer my original question... i agree that since state transfer is verifying block against orderer signature, that there is no chance of corrupted/tampered data bleeding into other peers via state transfer
Rachitga (Wed, 26 Jul 2017 08:41:49 GMT):
Has joined the channel.
Rachitga (Wed, 26 Jul 2017 08:43:27 GMT):
Hello All, I wanted to know if in the present architecture a malicious orderer is handled or not. Lets say an orderer service node is malicious and drops transactions and forms false blocks, the committers connected to this orderer have difference in the ledgers maintained by them. Do the peers then get the correct ledger by assuming a gossip protocol between them or gossip only happens when the peers are out of sync.
yacovm (Wed, 26 Jul 2017 08:54:03 GMT):
there is no byzantine support with regard to orderers
yacovm (Wed, 26 Jul 2017 08:54:04 GMT):
at v1.0
Rachitga (Wed, 26 Jul 2017 08:57:52 GMT):
okay so its not byzantine fault tolerant, but is it crash fault tolerant?
Rachitga (Wed, 26 Jul 2017 08:57:52 GMT):
@yacovm , okay so its not byzantine fault tolerant, but is it crash fault tolerant?
yacovm (Wed, 26 Jul 2017 09:09:11 GMT):
What is?
Rachitga (Wed, 26 Jul 2017 09:17:13 GMT):
committers
yacovm (Wed, 26 Jul 2017 09:30:17 GMT):
so it depends on the aspect
scottz (Wed, 26 Jul 2017 15:15:55 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=a6aZPkJgh32SdmNo9) @dave.enyeart What if someone spoofs the identity of an orderer, wtih a man-in-the-middle attack? (Should the peer really trust the orderer, without validation of hashes, etc?)
yacovm (Wed, 26 Jul 2017 15:34:16 GMT):
you mean MITM and impersonate the orderer @scottz ?
scottz (Wed, 26 Jul 2017 15:34:40 GMT):
yes
yacovm (Wed, 26 Jul 2017 15:37:18 GMT):
we have TLS in place
yacovm (Wed, 26 Jul 2017 15:37:29 GMT):
and the root CA certs are loaded from the config block
yacovm (Wed, 26 Jul 2017 15:37:31 GMT):
should be impossible
yacovm (Wed, 26 Jul 2017 15:37:41 GMT):
unless there is a bug in the code
qsmen (Fri, 28 Jul 2017 09:03:11 GMT):
if there are only two members in a channel and the two members are behind two different NATs respectively without public IP, it would be very difficult for them to form the same ledger.
qsmen (Fri, 28 Jul 2017 09:03:11 GMT):
if there are only two members in a channel and the two members are behind two different NATs respectively without public IP, is it difficult for them to form the same ledger?
qsmen (Fri, 28 Jul 2017 09:05:19 GMT):
it is very difficult for them to run the gossip protocol
qsmen (Fri, 28 Jul 2017 09:14:50 GMT):
maybe it is impossible to set up such a channel
qsmen (Fri, 28 Jul 2017 09:15:55 GMT):
IPV
qsmen (Fri, 28 Jul 2017 09:15:55 GMT):
IPV6 is greatly needed
yacovm (Fri, 28 Jul 2017 09:34:29 GMT):
how would IPv6 help?
qsmen (Fri, 28 Jul 2017 09:37:45 GMT):
if every peer has ipv6 address, then the gossip will run, then no problem
yacovm (Fri, 28 Jul 2017 09:45:23 GMT):
but he is behind a NAT
qsmen (Fri, 28 Jul 2017 09:46:44 GMT):
Your port forwarding is also a good method if I can configure the NAT. @yacovm
qsmen (Fri, 28 Jul 2017 09:47:53 GMT):
you mean if a computer is behind a NAT, it is impossible to have a public IP?
yacovm (Fri, 28 Jul 2017 09:48:04 GMT):
yes
qsmen (Fri, 28 Jul 2017 09:48:08 GMT):
ok
qsmen (Fri, 28 Jul 2017 09:48:14 GMT):
i see.
yacovm (Fri, 28 Jul 2017 09:48:19 GMT):
i need to go, bye
qsmen (Fri, 28 Jul 2017 09:48:28 GMT):
thank you
qsmen (Fri, 28 Jul 2017 09:48:32 GMT):
byebye
qizhang (Sun, 30 Jul 2017 21:34:23 GMT):
Hi, is there any function that related with a peer fetching a block from its neighbors? Thanks!
C0rWin (Sun, 30 Jul 2017 22:44:53 GMT):
@qizhang there is no direct API to fetch a block from other peers. there is a logic which detects a gap and request missing block based on ledger height advertised by channel peers.
C0rWin (Sun, 30 Jul 2017 22:45:21 GMT):
there are two type of messages responsible for that
C0rWin (Sun, 30 Jul 2017 22:45:36 GMT):
```// State transfer
// RemoteStateRequest is used to ask a set of blocks
// from a remote peer
message RemoteStateRequest {
uint64 start_seq_num = 1;
uint64 end_seq_num = 2;
}
// RemoteStateResponse is used to send a set of blocks
// to a remote peer
message RemoteStateResponse {
repeated Payload payloads = 1;
}
```
qizhang (Mon, 31 Jul 2017 18:27:38 GMT):
Thanks @C0rWin
berserkr (Wed, 02 Aug 2017 03:49:34 GMT):
Hi Guys, one question, on a node failure, how does gossip request the missing data from peers?
berserkr (Wed, 02 Aug 2017 03:50:32 GMT):
and if a block is fetched, does it validate anything? or does it just apply the changes in the write sets?
yacovm (Wed, 02 Aug 2017 05:17:18 GMT):
@berserkr it knows its own ledger height and other peers ledge height
yacovm (Wed, 02 Aug 2017 05:17:37 GMT):
It verifiee the signature of the orderer on the block
dave.enyeart (Wed, 02 Aug 2017 10:49:48 GMT):
@qizhang PayloadsBuffer is a gossip artifact, I'm re-posting your question over here
dave.enyeart (Wed, 02 Aug 2017 10:49:53 GMT):
>Hi, is the "PayloadsBuffer" a peer local buffer that stores the blocks before committing them to the ledger? Do these blocks include those received from either the orderer or the other peers? Thanks!
yacovm (Wed, 02 Aug 2017 10:51:26 GMT):
Nice classification Dave.
@qizhang yes both from the orderer and from peers
qizhang (Wed, 02 Aug 2017 14:06:43 GMT):
Thanks @dave.enyeart @yacovm .
Senthil1 (Sat, 05 Aug 2017 10:28:48 GMT):
Given that I have multiple ordering nodes in an ordering service (with a single kafka-zookeeper cluster), when i create a channel and make the peer join the channel, will gossip connects to all orderers in the ordering service for receiving the block or only one chosen at random or load balanced?
yacovm (Sat, 05 Aug 2017 10:40:39 GMT):
chosen at random
yacovm (Sat, 05 Aug 2017 10:41:13 GMT):
when disconnects - chosen at random again.
when receives a bad status from the orderer - chosen at random out of the other orderer nodes
Senthil1 (Sat, 05 Aug 2017 14:54:11 GMT):
thank @yacovm
Senthil1 (Sat, 05 Aug 2017 14:54:11 GMT):
thanks @yacovm
dave.enyeart (Sat, 05 Aug 2017 15:11:42 GMT):
@MeenakshiSingh Moving your question here...
MeenakshiSingh (Sat, 05 Aug 2017 15:11:42 GMT):
Has joined the channel.
dave.enyeart (Sat, 05 Aug 2017 15:11:47 GMT):
>I have a couple of doubts: 1. What exactly does the environment variable `CORE_PEER_GOSSIP_BOOTSTRAP` does? Does it enable gossip only between peers belonging to the same organization, and if so, if there are 4 peers within the same org, does each have to have a comma separated ip address list of all other peers in `CORE_PEER_GOSSIP_BOOTSTRAP`. A link/reference to documentation with all the env variables for 1.0 with explanation would be helpful. 2. What are anchor peers and what are their roles. Does the `CORE_PEER_GOSSIP_ORGLEADER` set the anchor peer?
yacovm (Sat, 05 Aug 2017 15:52:18 GMT):
So the bootstrap is a list of peers in the org that the peer knows. It uses them as a bootstrapping for membership.
yacovm (Sat, 05 Aug 2017 15:52:54 GMT):
if there are 4 peers within the same org - so they need to have a bootstrap list of some of the peers in the org, such that the transitive closure would be the entire group of 4 peers
yacovm (Sat, 05 Aug 2017 15:53:35 GMT):
i.e `p0,p1,p2,p3` you would need something like `p0: p1, p2: p1, p3: p1` etc. but not - `p0: p1, p1: p0, p2:p3, p3:p2`
yacovm (Sat, 05 Aug 2017 15:53:53 GMT):
anchor peers are peers from any org that participate in the channel
yacovm (Sat, 05 Aug 2017 15:54:04 GMT):
they are in a way - bootstrap peers for cross organization
yacovm (Sat, 05 Aug 2017 15:54:17 GMT):
the orgLeader variable doesn't set the anchor peer - it is set via the config block
yacovm (Sat, 05 Aug 2017 15:54:28 GMT):
@MeenakshiSingh ^
MeenakshiSingh (Sat, 05 Aug 2017 16:52:59 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=t7fhSfkjgkbPCzzZQ) @yacovm Ok..So, this means that for communication within the organization peers use gossip and nominated/representative (anchor peers) from each organization participate in communication via channel?
yacovm (Sat, 05 Aug 2017 17:01:55 GMT):
No.
yacovm (Sat, 05 Aug 2017 17:02:11 GMT):
The communication is p2p regardless of the anchor / bootstrap peers
yacovm (Sat, 05 Aug 2017 17:02:20 GMT):
they are just there to let the peers "know each other"
MeenakshiSingh (Sat, 05 Aug 2017 17:07:21 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=iL2qx4jbjgxEFWGrw) @yacovm so multiple peers can become anchor peers and participate in channels from an organization?
yacovm (Sat, 05 Aug 2017 17:07:36 GMT):
yes
MeenakshiSingh (Sun, 06 Aug 2017 05:19:31 GMT):
Hi..I am trying to run 2 peers on the same node. I generated the required crypto-materials with fabric-ca and kept in respective folders in mspconfig directory. Upon running the compose file, I am getting the following ```peer1.org2.example.com | 2017-08-06 04:58:14.008 UTC [flogging] setModuleLevel -> DEBU 1c4 Module 'grpc' logger enabled for log level 'ERROR'
peer1.org2.example.com | 2017-08-06 04:58:14.009 UTC [nodeCmd] func5 -> INFO 1c5 Starting profiling server with listenAddress = 0.0.0.0:6060
peer2.org2.example.com | 2017-08-06 04:58:14.100 UTC [gossip/comm] authenticateRemotePeer -> WARN 1c6 Identity store rejected 35.176.134.190:7051 : Peer Identity [0a 07 4f 72 67 32 4d 53 50 12 fe 06 2d 2d 2d 2d 2d 42 45 47 49 4e 20 2d 2d 2d 2d 2d 0a 4d 49 49 43 64 6a 43 43 41 68 32 67 41 77 49 42 41 67 49 55 42 4b 57 42 78 61 41 61 50 53 50 50 70 58 76 69 64 64 31 73 6b 45 55 51 64 71 6f 77 43 67 59 49 4b 6f 5a 49 7a 6a 30 45 41 77 49 77 0a 67 5a 73 78 43 7a 41 4a 42 67 4e 56 42 41 59 54 41 6c 56 54 4d 52 6b 77 46 77 59 44 56 51 51 49 45 78 42 4f 62 33 4a 30 61 43 42 44 59 57 78 70 5a 6d 39 79 62 6d 6c 68 4d 52 45 77 44 77 59 44 0a 56 51 51 48 45 77 68 54 59 57 34 67 53 47 39 7a 5a 54 45 67 4d 42 34 47 41 31 55 45 43 68 4d 58 53 47 6c 30 59 57 4e 6f 61 53 42 42 62 57 56 79 61 57 4e 68 49 45 78 70 62 57 6c 30 5a 57 51 78 0a 49 54 41 66 42 67 4e 56 42 41 73 54 47 46 4a 6c 63 32 56 68 63 6d 4e 6f 49 45 46 75 5a 47 41 31 55 45 43 68 4d 4c 0a 53 48 6c 77 5a 58 4a 73 5a 57 52 6e 5a 58 49 78 44 7a 41 4e 42 67 4e 56 42 41 73 54 42 6b 5a 68 59 6e 4a 70 59 7a 45 4f 4d 41 77 47 41 31 55 45 41 78 4d 46 63 47 56 6c 63 6a 45 77 57 54 41 54 0a 42 67 63 71 68 6b 6a 4f 50 51 49 42 42 67 67 71 68 6b 6a 4f 50 51 4d 42 42 77 4e 43 41 41 53 4f 45 54 48 62 62 69 41 32 41 46 4e 66 51 49 4d 72 2b 61 65 45 75 39 6d 47 4f 78 64 34 32 70 62 6c 0a 56 54 53 42 2b 59 52 36 6b 65 39 36 72 57 57 31 56 38 69 55 6e 2b 75 36 6e 66 4b 55 45 34 36 37 77 36 64 31 4f 48 74 6f 65 71 71 36 4b 4f 32 62 2b 68 6b 6c 6f 33 77 77 65 6a 41 4f 42 67 4e 56 0a 48 51 38 42 41 66 38 45 42 41 4d 43 42 34 41 77 44 41 59 44 56 52 30 54 41 51 48 2f 42 41 49 77 41 44 41 64 42 67 4e 56 48 51 34 45 46 67 51 55 49 4c 31 33 34 59 36 77 44 4a 4b 56 6c 77 39 59 0a 53 76 59 67 4f 6b 63 44 76 4b 63 77 48 77 59 44 56 52 30 6a 42 42 67 77 46 6f 41 55 43 79 2f 74 76 53 45 70 62 65 75 38 46 47 74 47 65 70 4b 74 6c 48 58 78 6a 65 30 77 47 67 59 44 56 52 30 52 0a 42 42 4d 77 45 59 49 50 61 58 41 74 4d 54 63 79 4c 54 4d 78 4c 54 45 79 4c 54 67 77 4d 41 6f 47 43 43 71 47 53 4d 34 39 42 41 4d 43 41 30 63 41 4d 45 51 43 49 48 4b 47 36 75 4c 44 49 2f 6e 61 0a 30 69 6b 6d 63 2b 49 41 72 31 65 4c 6c 6d 51 7a 63 44 35 44 52 5a 42 49 31 69 67 4c 52 76 77 53 41 69 41 35 30 64 39 42 6c 36 76 39 42 33 68 59 57 33 39 56 75 51 78 33 42 32 55 5a 31 4d 48 66 0a 56 69 37 4e 58 64 45 32 75 59 6e 2b 61 77 3d 3d 0a 2d 2d 2d 2d 2d 45 4e 44 20 2d 2d 2d 2d 2d 0a] cannot be validated. No MSP found able to do that.
peer2.org2.example.com | 2017-08-06 04:58:14.100 UTC [gossip/comm] Handshake -> WARN 1c7 Authentication failed: Peer Identity [0a 07 4f 72 67 32 4d 53 50 12 fe 06 2d 2d 2d 2d 2d 42 45 47 49 4e 20 2d 2d 2d 2d 2d 0a 4d 49 49 43 64 6a 43 43 41 68 32 67 41 77 49 42 41 67 49 55 42 4b 57 42 78 61 41 61 50 53 50 50 70 58 76 69 64 64 31 73 6b 45 55 51 64 71 6f 77 43 67 59 49 4b 6f 5a 49 7a 6a 30 45 41 77 49 77 0a 67 5a 73 78 43 7a 41 4a 42 67 4e 56 42 41 59 54 41 6c 56 54 4d 52 6b 77 46 77 59 44 56 51 51 49 45 78 42 4f 62 33 4a 30 61 43 42 44 59 57 78 70 5a 6d 39 79 62 6d 6c 68 4d 52 45 77 44 77 59 44 0a 56 51 51 48 45 77 68 54 59 57 34 67 53 47 39 7a 5a 54 45 67 4d 42 34 47 41 31 55 45 43 68 4d 58 53 47 6c 30 59 57 4e 6f 61 53 42 42 62 57 56 79 61 57 4e 68 49 45 78 70 62 57 6c 30 5a 57 51 78 0a 49 54 41 66 42 67 4e 56 42 41 73 54 47 46 4a 6c 63 32 56 68 63 6d 4e 6f 49 45 46 75 5a 43 42 45 5a 58 5a 6c 62 47 39 77 62 57 56 75 64 44 45 5a 4d 42 63 47 41 31 55 45 41 78 4d 51 5a 6d 46 69 0a 63 6d 6c 6a 4c 57 4e 68 4c 58 4e 6c 63 6e 5a 6c 63 6a 41 65 46 77 30 78 4e 7a 41 34 4d 44 55 78 4e 6a 51 77 4d 44 42 61 46 77 30 78 4f 44 41 34 4d 44 55 78 4e 6a 51 77 4d 44 42 61 4d 46 30 78 0a 43 7a 41 4a 42 67 4e 56 42 41 59 54 41 6c 56 54 4d 52 63 77 46 51 59 44 56 51 51 49 45 77 35 4f 62 33 4a 30 61 43 42 44 59 58 4a 76 62 47 6c 75 59 54 45 55 4d 42 49 47 41 31 44 49 2f 6e 61 0a 30 69 6b 6d 63 2b 49 41 72 31 65 4c 6c 6d 51 7a 63 44 35 44 52 5a 42 49 31 69 67 4c 52 76 77 53 41 69 41 35 30 64 39 42 6c 36 76 39 42 33 68 59 57 33 39 56 75 51 78 33 42 32 55 5a 31 4d 48 66 0a 56 69 37 4e 58 64 45 32 75 59 6e 2b 61 77 3d 3d 0a 2d 2d 2d 2d 2d 45 4e 44 20 2d 2d 2d 2d 2d 0a] cannot be validated. No MSP found able to do that.```
yacovm (Sun, 06 Aug 2017 05:40:21 GMT):
This is a setup problem
nnao (Sun, 06 Aug 2017 05:52:38 GMT):
Has joined the channel.
MeenakshiSingh (Sun, 06 Aug 2017 06:17:23 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=kH5teDiFY8TiixPHP) @yacovm You mean in certificate generation? After starting the peers I did `docker exec –i –t
yacovm (Sun, 06 Aug 2017 06:55:33 GMT):
are the peers in the same channel?
yacovm (Sun, 06 Aug 2017 06:55:35 GMT):
in the same org?
yacovm (Sun, 06 Aug 2017 06:55:40 GMT):
more info would help
rohitrocket (Sun, 06 Aug 2017 06:56:33 GMT):
Has joined the channel.
MeenakshiSingh (Sun, 06 Aug 2017 07:05:07 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=dd29pfTLK5NwrHjG2) @yacovm the peers are in the same organization. I haven't created the channel yet. Just setup the fabric-ca, generated the certificates, kept them in mspconfig directory structure and started the docker-compose. Here is the content of my docker-compose file: #
# Copyright IBM Corp All Rights Reserved
#
# SPDX-License-Identifier: Apache-2.0
#
version: '2'
networks:
basic:
services:
peer1.org2.example.com:
container_name: peer1.org2.example.com
image: hyperledger/fabric-peer:x86_64-1.0.0
environment:
- CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
- CORE_PEER_ID=peer1.org2.example.com
- CORE_PEER_ADDRESSAUTODETECT=false
- CORE_PEER_ENDORSER_ENABLED=true
- CORE_PEER_TLS_ENABLED=false
- CORE_PEER_GOSSIP_ORGLEADER=false
- CORE_PEER_GOSSIP_USELEADERELECTION=true
- CORE_PEER_PROFILE_ENABLED=true
- CORE_PEER_GOSSIP_EXTERNALENDPOINT=X.X.X.190:7051
- CORE_LOGGING_PEER=debug
- CORE_CHAINCODE_LOGGING_LEVEL=DEBUG
- CORE_PEER_LOCALMSPID=Org2MSP
- CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/peer/
- CORE_PEER_ADDRESS=X.X.X.190:7051
# # the following setting starts chaincode containers on the same
# # bridge network as the peers
# # https://docs.docker.com/compose/networking
- CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=${COMPOSE_PROJECT_NAME}_basic
- CORE_LEDGER_STATE_STATEDATABASE=CouchDB
- CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdb1:5984
working_dir: /opt/gopath/src/github.com/hyperledger/fabric
command: peer node start
# command: peer node start --peer-chaincodedev=true
ports:
- 7051:7051
- 7053:7053
volumes:
#- /var/run/:/host/var/run/
#- ./crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/msp:/etc/hyperledger/msp/peer
- /home/ubuntu/crypto-material/Org2/Peer1/mspconfig:/etc/hyperledger/msp/peer
#- ./crypto-config/peerOrganizations/org1.example.com/users:/etc/hyperledger/msp/users
#- ./config:/etc/hyperledger/configtx
depends_on:
#- orderer.example.com
- couchdb1
networks:
- basic
peer2.org2.example.com:
container_name: peer2.org2.example.com
image: hyperledger/fabric-peer:x86_64-1.0.0
environment:
- CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
- CORE_PEER_ID=peer2.org2.example.com
- CORE_PEER_ADDRESSAUTODETECT=false
- CORE_PEER_ENDORSER_ENABLED=true
- CORE_PEER_TLS_ENABLED=false
- CORE_PEER_GOSSIP_ORGLEADER=false
- CORE_PEER_GOSSIP_USELEADERELECTION=true
- CORE_PEER_PROFILE_ENABLED=true
- CORE_LOGGING_PEER=debug
- CORE_CHAINCODE_LOGGING_LEVEL=DEBUG
- CORE_PEER_LOCALMSPID=Org1MSP
- CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/peer/
- CORE_PEER_ADDRESS=X.X.X.190:8051
- CORE_PEER_GOSSIP_BOOTSTRAP=X.X.X.190:7051
# # the following setting starts chaincode containers on the same
# # bridge network as the peers
# # https://docs.docker.com/compose/networking
- CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=${COMPOSE_PROJECT_NAME}_basic
- CORE_LEDGER_STATE_STATEDATABASE=CouchDB
- CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdb2:5984
working_dir: /opt/gopath/src/github.com/hyperledger/fabric
command: peer node start
# command: peer node start --peer-chaincodedev=true
ports:
- 8051:8051
- 8053:8053
volumes:
#- /var/run/:/host/var/run/
#- ./crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/msp:/etc/hyperledger/msp/peer
- /home/ubuntu/crypto-material/Org2/Peer2/mspconfig:/etc/hyperledger/msp/peer
#- ./crypto-config/peerOrganizations/org1.example.com/users:/etc/hyperledger/msp/users
#- ./config:/etc/hyperledger/configtx
depends_on:
#- orderer.example.com
- couchdb2
networks:
- basic
couchdb1:
container_name: couchdb1
image: hyperledger/fabric-couchdb:x86_64-1.0.0
ports:
- 5984:5984
environment:
DB_URL: http://localhost:5984/member_db
networks:
- basic
couchdb2:
container_name: couchdb2
image: hyperledger/fabric-couchdb:x86_64-1.0.0
ports:
- 6984:5984
environment:
DB_URL: http://localhost:5984/member_db
networks:
- basic
yacovm (Sun, 06 Aug 2017 07:07:42 GMT):
I really can't understand anything this way
yacovm (Sun, 06 Aug 2017 07:07:48 GMT):
can you please describe the setup in words?
MeenakshiSingh (Sun, 06 Aug 2017 07:15:44 GMT):
I setup the fabric-ca server and client. I registered and enrolled 2 roles: admin and 2 peers viz, peer1 and peer2. This generated the public/private keys for each role and the cacerts. As per the documentation: https://media.readthedocs.org/pdf/hyperledger-fabric/latest/hyperledger-fabric.pdf --page 74, I created the mspconfig directory for each peer with admincerts, cacerts, keystore and signcerts as subdirectories. I copied the required certificates in the respective subdirectorires and the mentioned the path `/home/ubuntu/crypto-material/Org1/Peer1/mspconfig` for mounting it to the docker container. Then I started docker-compose.
yacovm (Sun, 06 Aug 2017 07:16:26 GMT):
so you have only 1 org?
MeenakshiSingh (Sun, 06 Aug 2017 07:37:04 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=ZgT67yAp35LfcZiWb) @yacovm yes
yacovm (Sun, 06 Aug 2017 07:41:18 GMT):
aha
yacovm (Sun, 06 Aug 2017 07:41:31 GMT):
so if that doesn't work - it means you have a bad setup of the stuff inside the MSPDir
yacovm (Sun, 06 Aug 2017 07:41:34 GMT):
in any case
yacovm (Sun, 06 Aug 2017 07:41:38 GMT):
that's not a gossip issue :(
yacovm (Sun, 06 Aug 2017 07:41:44 GMT):
The peers don't know each other
yacovm (Sun, 06 Aug 2017 07:41:44 GMT):
The peers don't know each other and when they connect to each other
yacovm (Sun, 06 Aug 2017 07:41:50 GMT):
and can't verify each other's certificate
yacovm (Sun, 06 Aug 2017 07:42:15 GMT):
I think there is a misconfiguration
yacovm (Sun, 06 Aug 2017 07:42:30 GMT):
ah wait
yacovm (Sun, 06 Aug 2017 07:42:35 GMT):
if its the same org
yacovm (Sun, 06 Aug 2017 07:42:41 GMT):
why do you have there - 2 different MSPIDs?
yacovm (Sun, 06 Aug 2017 07:42:51 GMT):
`- CORE_PEER_LOCALMSPID=Org1MSP` and `- CORE_PEER_LOCALMSPID=Org2MSP` ?
MeenakshiSingh (Sun, 06 Aug 2017 07:49:28 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=id2shrBAwk4rEFeu4) @yacovm srry...m still very new to this. Nice catch!..changed the mspid to org2 and its working.. Thanks a lot :)
y204990 (Sun, 06 Aug 2017 16:25:36 GMT):
Has joined the channel.
rahulhegde (Sun, 06 Aug 2017 17:46:28 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=CRacRxjGDEuCZgb5C) @yacovm
Does this declaration indicates ` p0 ` is always the Gossip Organization Leader?
yacovm (Sun, 06 Aug 2017 18:37:25 GMT):
no
sh (Sun, 06 Aug 2017 22:22:29 GMT):
Has joined the channel.
AdnanC (Mon, 07 Aug 2017 16:38:57 GMT):
A question regarding gossip non-leader: if a peer is set up as a) non-leader and b) no leader election, c) and the leader peer is online
will the non-leader peer ever get blocks from the orderer directly?
AdnanC (Mon, 07 Aug 2017 16:50:58 GMT):
I would guess the answer is no, but I see in some cases (if the non-leader was down for some time and comes back up) it gets blocks from the orderer
yacovm (Mon, 07 Aug 2017 16:53:14 GMT):
impossible
C0rWin (Mon, 07 Aug 2017 17:32:04 GMT):
@AdnanC does non leader peer have bootstrap set of peers?
C0rWin (Mon, 07 Aug 2017 17:32:31 GMT):
Or is there anchor peers in the channel?
AdnanC (Mon, 07 Aug 2017 19:03:22 GMT):
@C0rWin it has a bootstrap peer (which is the same as the selected leader) and the anchor peer is also the same
C0rWin (Mon, 07 Aug 2017 19:03:53 GMT):
This is why it get in sync then
AdnanC (Mon, 07 Aug 2017 19:04:31 GMT):
you mean it gets blocks from the leader?
C0rWin (Mon, 07 Aug 2017 19:04:31 GMT):
It builds gossip membership and leader forwards block to it
C0rWin (Mon, 07 Aug 2017 19:04:41 GMT):
Yes, exactly
AdnanC (Mon, 07 Aug 2017 19:04:52 GMT):
That was my expectation too
AdnanC (Mon, 07 Aug 2017 19:05:04 GMT):
but I corrupted the blocks i the leader
AdnanC (Mon, 07 Aug 2017 19:05:21 GMT):
but it still ends up with correct blocks
AdnanC (Mon, 07 Aug 2017 19:05:21 GMT):
but the non-leader still ends up with correct blocks
AdnanC (Mon, 07 Aug 2017 19:05:45 GMT):
I observe that if I disconnect the orderer, it does not get the blocks anymore
AdnanC (Mon, 07 Aug 2017 19:06:07 GMT):
so here's the flow of operations: .....
AdnanC (Mon, 07 Aug 2017 19:09:17 GMT):
1. spin up 4 peer, 2 org network where the lower numbered peers are "selected" as leader and gossip elections is turned off for all
2. After some time, I stop non-leader of org1 and both the peers of org2
3. 5 transactions in org1 leader
5. Manually corrupt the ledgerfile in org1 leader
4. restart the non-leader of the org1
5. Query the nonleader
I expected that non-leader will not have the full ledger, but it has
AdnanC (Mon, 07 Aug 2017 19:09:17 GMT):
1. spin up 4 peer, 2 org network where the lower numbered peers are "selected" as leader and gossip elections is turned off for all
2. After some time, I stop non-leader of org1 and both the peers of org2
3. 5 transactions in org1 leader
5. Manually corrupt the ledgerfile in org1 leader
4. restart the non-leader of the org1
5. Query the nonleader
I expected that non-leader will *not* have the full ledger, as it is supposed to get them from its leader's ledger (which is corrupted)but I observe that the non-leader has all the correct blocks
AdnanC (Mon, 07 Aug 2017 19:09:46 GMT):
if I stop the orderer, then I see that non-leader has the expected, curtailed ledger
AdnanC (Mon, 07 Aug 2017 19:09:46 GMT):
if I stop the ordere before non-leader comes back up, then I see that non-leader has the expected, curtailed ledger
AdnanC (Mon, 07 Aug 2017 19:09:46 GMT):
if I stop the orderer before non-leader comes back up, then I see that non-leader has the expected, curtailed ledger
AdnanC (Mon, 07 Aug 2017 19:09:46 GMT):
if I stop the orderer before non-leader comes back up(i,e. before step 4), then I see that non-leader has the expected, curtailed ledger @C0rWin
C0rWin (Mon, 07 Aug 2017 19:19:14 GMT):
Interesting
C0rWin (Mon, 07 Aug 2017 19:19:45 GMT):
Could you please report JIRA and tag me in?
AdnanC (Mon, 07 Aug 2017 19:20:53 GMT):
sure. Is there a easy way to verify (maybe from the logs) that a peer is actually getting blocks from the the orderer?
yacovm (Mon, 07 Aug 2017 19:25:41 GMT):
I guess maybe the leader still has the blocks in its memory
yacovm (Mon, 07 Aug 2017 19:25:48 GMT):
and the non leader that comes alive
yacovm (Mon, 07 Aug 2017 19:25:51 GMT):
receives them via pull
yacovm (Mon, 07 Aug 2017 19:26:00 GMT):
before it receives them via state transfer?
C0rWin (Mon, 07 Aug 2017 19:26:09 GMT):
Yes, that's possible
yacovm (Mon, 07 Aug 2017 19:26:11 GMT):
we can fetch up to 100 blocks at a time IIRC
yacovm (Mon, 07 Aug 2017 19:26:16 GMT):
so 5 blocks won't cut it ;)
C0rWin (Mon, 07 Aug 2017 19:26:42 GMT):
We have expiration so can check it easily
yacovm (Mon, 07 Aug 2017 19:26:48 GMT):
also - I'm not an expert on levelDB but from what I know - LSM tree-type DBs have a level 0 that is stored in memory. it's usually a few tens of MBs
yacovm (Mon, 07 Aug 2017 19:27:02 GMT):
yep
yacovm (Mon, 07 Aug 2017 19:27:17 GMT):
he just needs to wait... how many time was it again? :thinking:
C0rWin (Mon, 07 Aug 2017 19:27:44 GMT):
Need to look in the code, connected from mobile
yacovm (Mon, 07 Aug 2017 19:28:45 GMT):
should be like 10 min @AdnanC
AdnanC (Mon, 07 Aug 2017 19:29:12 GMT):
from step1 to step 5 is like maybe 2 minutes
yacovm (Mon, 07 Aug 2017 19:29:27 GMT):
Fabric is full of so many mechanisms, so hard to test :grin:
AdnanC (Mon, 07 Aug 2017 19:30:09 GMT):
A question, how does leveldb come in play here? Gossip transfers only ledger blocks, not the statedb things, right?
AdnanC (Mon, 07 Aug 2017 19:30:09 GMT):
A question, I am wondering how does leveldb come in play here... Gossip transfers only ledger blocks, not the statedb things, right?
yacovm (Mon, 07 Aug 2017 19:31:38 GMT):
ah right I forgot
yacovm (Mon, 07 Aug 2017 19:31:47 GMT):
yeah yeah
yacovm (Mon, 07 Aug 2017 19:31:49 GMT):
you're right, my bad
yacovm (Mon, 07 Aug 2017 19:32:42 GMT):
My head is in sideDB mode from CRing something
AdnanC (Mon, 07 Aug 2017 19:32:53 GMT):
ok :)
smcambria22 (Mon, 07 Aug 2017 19:58:37 GMT):
Has joined the channel.
AdnanC (Mon, 07 Aug 2017 21:39:43 GMT):
@C0rWin https://jira.hyperledger.org/browse/FAB-5651
yacovm (Mon, 07 Aug 2017 21:50:58 GMT):
```
[36m2017-08-07 21:30:19.084 UTC [gossip/gossip] handleMessage -> DEBU af8[0m Entering, 172.18.0.5:7051 [5 44 185 46 104 194 176 45 64 104 113 115 124 27 109 10 119 69 173 36 41 135 147 41 40 98 34 138 243 34 17 7] sent us GossipMessage: Channel: [98 101 104 97 118 101 115 121 115 116 101 115 116], nonce: 0, tag: CHAN_AND_ORG DataUpdate: Type: BLOCK_MSG, items: 4, nonce: 5077589437772028536, Envelope: 19533 bytes, Signature: 0 bytes
```
yacovm (Mon, 07 Aug 2017 21:50:58 GMT):
```
[36m2017-08-07 21:30:19.084 UTC [gossip/gossip] handleMessage -> DEBU af8[0m Entering, 172.18.0.5:7051 [5 44 185 46 104 194 176 45 64 104 113 115 124 27 109 10 119 69 173 36 41 135 147 41 40 98 34 138 243 34 17 7] sent us GossipMessage: Channel: [98 101 104 97 118 101 115 121 115 116 101 115 116], nonce: 0, tag: CHAN_AND_ORG DataUpdate: Type: BLOCK_MSG, items: 4, nonce: 5077589437772028536, Envelope: 19533 bytes, Signature: 0 bytes
```
yacovm (Mon, 07 Aug 2017 21:51:20 GMT):
this is from `non-leader-log.txt`
Hangyu (Tue, 08 Aug 2017 09:00:36 GMT):
Has joined the channel.
qizhang (Tue, 08 Aug 2017 18:34:44 GMT):
I read the following example from *Problem 3* section in this document https://docs.google.com/document/d/1vNMaM7XhOlu9tB_10dKnlrhy5d7b1u8lSY8a-kVjCO4/edit , but not quite understand it. '''*Consider now the case where you have a batchTimeout of 1 second, and two OSNs. A batch has just been cut and a new transaction comes in via, say, OSN1. It gets posted to the partition. OSN2 reads it at time t=5s and sets a timer that will fire at t=6s. OSN1 reads this transaction from the partition at time t=5.6s and sets its own timer accordingly. A second transaction is posted to the partition and is read by OSN2 at t=6.2s, and by OSN1 at t=6.5s. We are now in a situation where the OSN1 has cut a block with both of these transactions, whereas OSN2 cut a block with just the first of them. The two OSNs have now diverged and are outputting different sequences of blocks — this is unacceptable.*'''
qizhang (Tue, 08 Aug 2017 18:34:44 GMT):
I read the following example from *Problem 3* section in this document https://docs.google.com/document/d/1vNMaM7XhOlu9tB_10dKnlrhy5d7b1u8lSY8a-kVjCO4/edit , but not quite understand it. "Consider now the case where you have a batchTimeout of 1 second, and two OSNs. A batch has just been cut and a new transaction comes in via, say, OSN1. It gets posted to the partition. OSN2 reads it at time t=5s and sets a timer that will fire at t=6s. OSN1 reads this transaction from the partition at time t=5.6s and sets its own timer accordingly. A second transaction is posted to the partition and is read by OSN2 at t=6.2s, and by OSN1 at t=6.5s. We are now in a situation where the OSN1 has cut a block with both of these transactions, whereas OSN2 cut a block with just the first of them. The two OSNs have now diverged and are outputting different sequences of blocks — this is unacceptable"
qizhang (Tue, 08 Aug 2017 18:34:44 GMT):
I read the following example from *Problem 3* section in this document https://docs.google.com/document/d/1vNMaM7XhOlu9tB_10dKnlrhy5d7b1u8lSY8a-kVjCO4/edit , but not quite understand it. '''Consider now the case where you have a batchTimeout of 1 second, and two OSNs. A batch has just been cut and a new transaction comes in via, say, OSN1. It gets posted to the partition. OSN2 reads it at time t=5s and sets a timer that will fire at t=6s. OSN1 reads this transaction from the partition at time t=5.6s and sets its own timer accordingly. A second transaction is posted to the partition and is read by OSN2 at t=6.2s, and by OSN1 at t=6.5s. We are now in a situation where the OSN1 has cut a block with both of these transactions, whereas OSN2 cut a block with just the first of them. The two OSNs have now diverged and are outputting different sequences of blocks — this is unacceptable'''
qizhang (Tue, 08 Aug 2017 18:34:44 GMT):
I read the following example from *Problem 3* section in this document https://docs.google.com/document/d/1vNMaM7XhOlu9tB_10dKnlrhy5d7b1u8lSY8a-kVjCO4/edit , but not quite understand it. 'Consider now the case where you have a batchTimeout of 1 second, and two OSNs. A batch has just been cut and a new transaction comes in via, say, OSN1. It gets posted to the partition. OSN2 reads it at time t=5s and sets a timer that will fire at t=6s. OSN1 reads this transaction from the partition at time t=5.6s and sets its own timer accordingly. A second transaction is posted to the partition and is read by OSN2 at t=6.2s, and by OSN1 at t=6.5s. We are now in a situation where the OSN1 has cut a block with both of these transactions, whereas OSN2 cut a block with just the first of them. The two OSNs have now diverged and are outputting different sequences of blocks — this is unacceptable'
qizhang (Tue, 08 Aug 2017 18:34:44 GMT):
I read the following example from *Problem 3* section in this document https://docs.google.com/document/d/1vNMaM7XhOlu9tB_10dKnlrhy5d7b1u8lSY8a-kVjCO4/edit , but not quite understand it. \n'Consider now the case where you have a batchTimeout of 1 second, and two OSNs. A batch has just been cut and a new transaction comes in via, say, OSN1. It gets posted to the partition. OSN2 reads it at time t=5s and sets a timer that will fire at t=6s. OSN1 reads this transaction from the partition at time t=5.6s and sets its own timer accordingly. A second transaction is posted to the partition and is read by OSN2 at t=6.2s, and by OSN1 at t=6.5s. We are now in a situation where the OSN1 has cut a block with both of these transactions, whereas OSN2 cut a block with just the first of them. The two OSNs have now diverged and are outputting different sequences of blocks — this is unacceptable'
qizhang (Tue, 08 Aug 2017 18:34:44 GMT):
{I read the following example from *Problem 3* section in this document https://docs.google.com/document/d/1vNMaM7XhOlu9tB_10dKnlrhy5d7b1u8lSY8a-kVjCO4/edit , but not quite understand it. \n"Consider now the case where you have a batchTimeout of 1 second, and two OSNs. A batch has just been cut and a new transaction comes in via, say, OSN1. It gets posted to the partition. OSN2 reads it at time t=5s and sets a timer that will fire at t=6s. OSN1 reads this transaction from the partition at time t=5.6s and sets its own timer accordingly. A second transaction is posted to the partition and is read by OSN2 at t=6.2s, and by OSN1 at t=6.5s. We are now in a situation where the OSN1 has cut a block with both of these transactions, whereas OSN2 cut a block with just the first of them. The two OSNs have now diverged and are outputting different sequences of blocks — this is unacceptable"}
qizhang (Tue, 08 Aug 2017 18:34:44 GMT):
I read the following example from *Problem 3* section in this document https://docs.google.com/document/d/1vNMaM7XhOlu9tB_10dKnlrhy5d7b1u8lSY8a-kVjCO4/edit , but not quite understand it. "Consider now the case where you have a batchTimeout of 1 second, and two OSNs. A batch has just been cut and a new transaction comes in via, say, OSN1. It gets posted to the partition. OSN2 reads it at time t=5s and sets a timer that will fire at t=6s. OSN1 reads this transaction from the partition at time t=5.6s and sets its own timer accordingly. A second transaction is posted to the partition and is read by OSN2 at t=6.2s, and by OSN1 at t=6.5s. We are now in a situation where the OSN1 has cut a block with both of these transactions, whereas OSN2 cut a block with just the first of them. The two OSNs have now diverged and are outputting different sequences of blocks — this is unacceptable"
qizhang (Tue, 08 Aug 2017 18:34:44 GMT):
I read the following example from *Problem 3* section in this document https://docs.google.com/document/d/1vNMaM7XhOlu9tB_10dKnlrhy5d7b1u8lSY8a-kVjCO4/edit , but not quite understand it. `Consider now the case where you have a batchTimeout of 1 second, and two OSNs. A batch has just been cut and a new transaction comes in via, say, OSN1. It gets posted to the partition. OSN2 reads it at time t=5s and sets a timer that will fire at t=6s. OSN1 reads this transaction from the partition at time t=5.6s and sets its own timer accordingly. A second transaction is posted to the partition and is read by OSN2 at t=6.2s, and by OSN1 at t=6.5s. We are now in a situation where the OSN1 has cut a block with both of these transactions, whereas OSN2 cut a block with just the first of them. The two OSNs have now diverged and are outputting different sequences of blocks — this is unacceptable`
qizhang (Tue, 08 Aug 2017 18:34:44 GMT):
I read the following example from *Problem 3* section in this document https://docs.google.com/document/d/1vNMaM7XhOlu9tB_10dKnlrhy5d7b1u8lSY8a-kVjCO4/edit , but not quite understand it. ```Consider now the case where you have a batchTimeout of 1 second, and two OSNs. A batch has just been cut and a new transaction comes in via, say, OSN1. It gets posted to the partition. OSN2 reads it at time t=5s and sets a timer that will fire at t=6s. OSN1 reads this transaction from the partition at time t=5.6s and sets its own timer accordingly. A second transaction is posted to the partition and is read by OSN2 at t=6.2s, and by OSN1 at t=6.5s. We are now in a situation where the OSN1 has cut a block with both of these transactions, whereas OSN2 cut a block with just the first of them. The two OSNs have now diverged and are outputting different sequences of blocks — this is unacceptable```
qizhang (Tue, 08 Aug 2017 18:36:09 GMT):
I hope this is the correct place to ask, please let me know if it is not :-)
qizhang (Tue, 08 Aug 2017 18:38:12 GMT):
*OSN2 reads it at time t=5s and sets a timer that will fire at t=6s*, does it mean that OSN2 sets a timer when it reads a block from the partition?
yacovm (Tue, 08 Aug 2017 20:05:04 GMT):
the ordering service node has a batch timeout that defines the frequency of it cutting blocks
yacovm (Tue, 08 Aug 2017 20:05:19 GMT):
not sure about the implementation though
yacovm (Tue, 08 Aug 2017 20:05:30 GMT):
the right place to ask is #fabric-consensus
AdnanC (Tue, 08 Aug 2017 20:35:20 GMT):
A question regarding Gossip state transfers: Does a "pull" do state transfers too, or is it only for exchanging block recieve information etc? (reading the design docs)
C0rWin (Tue, 08 Aug 2017 20:55:39 GMT):
pull has nothing to do with a state transfer
C0rWin (Tue, 08 Aug 2017 20:56:27 GMT):
pull used to query for latest updates on neighbors sliding window of blocks
C0rWin (Tue, 08 Aug 2017 20:56:59 GMT):
while state transfer is actually querying the ledger and capable to replicate whole ledger
AdnanC (Tue, 08 Aug 2017 20:57:43 GMT):
Thanks
AdnanC (Tue, 08 Aug 2017 20:58:30 GMT):
And the State transfer, is it of two types, based on how large is the data transfer / or how long have a network been up before a peer joins?
C0rWin (Tue, 08 Aug 2017 21:02:44 GMT):
@AdnanC of two types? what do you mean?
AdnanC (Tue, 08 Aug 2017 21:04:21 GMT):
From the design document, section 1.2, it sounds like there are two types, one for peers that missed a couple blocks (due to network hiccup etc)\and another for peers that are joining a channel that has been up fore while
AdnanC (Tue, 08 Aug 2017 21:04:21 GMT):
From the design document, section 1.2, it sounds like there are two types, one for peers that missed a couple blocks (due to network hiccup etc)\and another for peers that are joining a channel that has been up for a while
AdnanC (Tue, 08 Aug 2017 21:04:21 GMT):
From the design document, section 1.2, it sounds like there are two types, one for peers that missed a couple blocks (due to network hiccup etc) and another for peers that are joining a channel that has been up for a while
C0rWin (Tue, 08 Aug 2017 21:06:47 GMT):
this are two use cases
C0rWin (Tue, 08 Aug 2017 21:06:53 GMT):
while the mechanism is the same
AdnanC (Tue, 08 Aug 2017 21:06:57 GMT):
Ah ok
MeenakshiSingh (Wed, 09 Aug 2017 06:51:09 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=jWYTGmWdCPm73eDJw) @AdnanC Can we manually tweak the couchDB to add or remove the transactions? If so, how is that possible?
C0rWin (Wed, 09 Aug 2017 07:29:04 GMT):
@MeenakshiSingh moving your question to #fabric-ledger as it's more appropriate channel to answer this question
SotirisAlfonsos (Wed, 09 Aug 2017 13:43:25 GMT):
Hello. I do not know if this is the right channel for this question but i can not find a better fit. On large numbers of transactions (1000tps, 4 peer endorsement), i get an ssl error, or connection failed from the client to the peers ( peers do not crash ). I assume that it might be because i reach the limit of ssl connections. Is my assumption true, or could i configure this number? ( i tried increasing the number of file descriptors but did not change anything)
yacovm (Wed, 09 Aug 2017 13:44:54 GMT):
this has nothing to do with gossip though
SotirisAlfonsos (Wed, 09 Aug 2017 13:48:10 GMT):
I know. It is not really related to any of the channels.
yacovm (Wed, 09 Aug 2017 13:51:06 GMT):
ah ok
yacovm (Wed, 09 Aug 2017 13:51:15 GMT):
that's interesting
yacovm (Wed, 09 Aug 2017 13:51:17 GMT):
by the way
yacovm (Wed, 09 Aug 2017 13:51:25 GMT):
do you remember that other error you reported about back then?
yacovm (Wed, 09 Aug 2017 13:51:28 GMT):
you opened a JIRA
yacovm (Wed, 09 Aug 2017 13:51:30 GMT):
remember
yacovm (Wed, 09 Aug 2017 13:51:31 GMT):
?
SotirisAlfonsos (Wed, 09 Aug 2017 13:52:42 GMT):
Yes
yacovm (Wed, 09 Aug 2017 13:52:50 GMT):
so I fixed it!
yacovm (Wed, 09 Aug 2017 13:52:54 GMT):
you can try and check
SotirisAlfonsos (Wed, 09 Aug 2017 13:53:10 GMT):
Yes i saw it was fixed :)
SotirisAlfonsos (Wed, 09 Aug 2017 13:53:44 GMT):
I have not managed to synchronize to release yet though
SotirisAlfonsos (Wed, 09 Aug 2017 13:58:18 GMT):
But after all the issue was on the leader election timer, or was the same peer being reelected? Just curious
yacovm (Wed, 09 Aug 2017 13:58:43 GMT):
there was no relinquishing of leadership
yacovm (Wed, 09 Aug 2017 13:58:50 GMT):
in case of unable to connect to the orderer
SotirisAlfonsos (Wed, 09 Aug 2017 14:00:02 GMT):
Ah i thought there was a timer for leader election.
SotirisAlfonsos (Wed, 09 Aug 2017 14:00:31 GMT):
So a peer remains the leader as long as he is connected
qsmen (Fri, 11 Aug 2017 05:53:12 GMT):
Hello， to gossip information within a channel, the peer should know the rest of peers in the channel. How does the peer know the others? Thank you
yacovm (Fri, 11 Aug 2017 05:54:13 GMT):
It knows the anchor peers
yacovm (Fri, 11 Aug 2017 05:54:28 GMT):
From the config block
qsmen (Fri, 11 Aug 2017 05:56:40 GMT):
it first send its own address to anchor peer, and get back other peer's address?
qsmen (Fri, 11 Aug 2017 05:57:32 GMT):
ok, thank you, Yacovm
yacovm (Fri, 11 Aug 2017 06:07:45 GMT):
Something like this, yes
qsmen (Fri, 11 Aug 2017 06:30:03 GMT):
ok, thank you again,Yacovm.
dnzdlklc (Fri, 11 Aug 2017 09:29:05 GMT):
Has joined the channel.
suryalanka (Mon, 14 Aug 2017 15:07:28 GMT):
Has joined the channel.
suryalanka (Mon, 14 Aug 2017 15:09:12 GMT):
Hi, what does the following warnings in the peer logs suggest? ```2017-08-11 21:03:28.213 UTC [blocksProvider] DeliverBlocks -> WARN 50cb6 Failed adding payload of 38279 because: Ledger height is at 31126, cannot enqueue block with sequence of 38279
2017-08-11 21:03:28.225 UTC [blocksProvider] DeliverBlocks -> WARN 50cb7 Failed adding payload of 38193 because: Ledger height is at 29737, cannot enqueue block with sequence of 38193
2017-08-11 21:03:28.246 UTC [blocksProvider] DeliverBlocks -> WARN 50cb8 Failed adding payload of 38295 because: Ledger height is at 30012, cannot enqueue block with sequence of 38295```
suryalanka (Mon, 14 Aug 2017 15:10:08 GMT):
We are seeing the above warnings in the endorser peers in the organization
yacovm (Mon, 14 Aug 2017 15:11:04 GMT):
That happens if you get blocks that are "too far"
yacovm (Mon, 14 Aug 2017 15:11:07 GMT):
from your ledger's height
yacovm (Mon, 14 Aug 2017 15:11:26 GMT):
in example- if your peer is *really behind* other peers
yacovm (Mon, 14 Aug 2017 15:11:43 GMT):
and they send your peer a block that is really really "in the future" with regard to your peer
yacovm (Mon, 14 Aug 2017 15:11:55 GMT):
then your peer - refuses to enqueue the block to memory and simply throws it away
yacovm (Mon, 14 Aug 2017 15:12:02 GMT):
(it will get the block eventually)
yacovm (Mon, 14 Aug 2017 15:12:14 GMT):
This is a protection mechanism so your memory won't explode ;)
yacovm (Mon, 14 Aug 2017 15:12:59 GMT):
By the way how did you know to ask in this channel @suryalanka ?
suryalanka (Mon, 14 Aug 2017 15:14:35 GMT):
we found `Ledger height is at 30012, cannot enqueue block with sequence of 38295` snippet in the gossip code (gossip/state/state.go)
yacovm (Mon, 14 Aug 2017 15:14:46 GMT):
ah, ok
suryalanka (Mon, 14 Aug 2017 15:19:02 GMT):
Thank you, how do I verify that the block is delivered?
yacovm (Mon, 14 Aug 2017 15:19:39 GMT):
you look at the logs
yacovm (Mon, 14 Aug 2017 15:19:46 GMT):
and you can also use events
yacovm (Mon, 14 Aug 2017 15:20:03 GMT):
you can subscribe to block events from `{node, go}-SDK`
yacovm (Mon, 14 Aug 2017 15:20:03 GMT):
you can subscribe to block events from `{node, go}SDK`
suryalanka (Mon, 14 Aug 2017 16:15:56 GMT):
thank you
scottz (Mon, 14 Aug 2017 16:26:59 GMT):
@yacov surya and I are still troubleshooting some peers connections to orderers - and we see 3 hours of nothing but that same long 'cannot enqueue block'. more on that later.
scottz (Mon, 14 Aug 2017 16:28:06 GMT):
A different question: when we start a network, peers choose randomly which orderer to connect for delivered blocks. However, if we stop the orderer that it is connected to, then (from a small test sample) it looks like a peer will always attempt to connect first to orderer0 (and then move in sequence looking through the rest of the orderers list, which is fine). It seems the peer should choose randomly. Otherwise, we could see all peers connected to the same orderer (for example after restarting all the orderers once). Would you agree?
yacovm (Mon, 14 Aug 2017 17:25:30 GMT):
How many are there?
yacovm (Mon, 14 Aug 2017 17:46:22 GMT):
> @yacov surya and I are still troubleshooting some peers connections to orderers - and we see 3 hours of nothing but that same long 'cannot enqueue block'. more on that later.
So you claim that the peer connects to the orderer, but emits this message?! @scottz
yacovm (Mon, 14 Aug 2017 17:46:50 GMT):
This shouldn't be possible, since for each block the peer pulls - it puts it into that payload buffer in-order
yacovm (Mon, 14 Aug 2017 17:47:35 GMT):
Maybe the peer is not a leader and gets the block from some other peer?
suryalanka (Mon, 14 Aug 2017 18:23:00 GMT):
@yacovm every peer in the org is org leader
suryalanka (Mon, 14 Aug 2017 18:23:00 GMT):
@yacovm every peer in the org is org leader in this case
yacovm (Mon, 14 Aug 2017 18:24:24 GMT):
More information can be useful
yacovm (Mon, 14 Aug 2017 18:24:28 GMT):
how many peers per org, etc.
yacovm (Mon, 14 Aug 2017 18:24:34 GMT):
is leader election on? etc. etc.
suryalanka (Mon, 14 Aug 2017 18:28:44 GMT):
each organization has 2 peers(one peer is endorser and other is committer). `CORE_PEER_GOSSIP_ORGLEADER` is set to `true` for both the peers.
yacovm (Mon, 14 Aug 2017 19:02:10 GMT):
Interesting...
yacovm (Mon, 14 Aug 2017 19:02:33 GMT):
And which peer emits these warnings?
yacovm (Mon, 14 Aug 2017 19:06:19 GMT):
What is the setup? Docker compose? What did you do to make it happen?
yacovm (Mon, 14 Aug 2017 19:06:43 GMT):
And... are you in the office in the next 2 hours or so?
yacovm (Mon, 14 Aug 2017 19:10:54 GMT):
@suryalanka
suryalanka (Mon, 14 Aug 2017 19:12:06 GMT):
@yacovm endorser is emitting those warnings
yacovm (Mon, 14 Aug 2017 19:12:49 GMT):
Only 1 of the peers?
suryalanka (Mon, 14 Aug 2017 19:12:57 GMT):
yes
yacovm (Mon, 14 Aug 2017 19:13:08 GMT):
Are both in the channel? Is he connected to the orderer?
suryalanka (Mon, 14 Aug 2017 19:14:13 GMT):
yes both the peers joined the channel... committer is getting the blocks, but not the endorser
suryalanka (Mon, 14 Aug 2017 19:14:53 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=sFcPys2yEzy6FqsTE) @yacovm yes, I will be available
yacovm (Mon, 14 Aug 2017 19:18:43 GMT):
I'll contact you when i get home, dont move :wink:
C0rWin (Mon, 14 Aug 2017 20:07:12 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=zA5xvgjpWS9ubMh6S) @suryalanka what is the rate? how is that possible that there is such a gap of almost 8k blocks?
C0rWin (Mon, 14 Aug 2017 20:07:31 GMT):
how mane printouts like these do you see?
C0rWin (Mon, 14 Aug 2017 20:07:48 GMT):
can you use QSCC and check ledger height at each peer?
berserkr (Mon, 14 Aug 2017 23:42:48 GMT):
@yacovm when a node fails and needs to use gossip to catch up, it will see the latest block size to be blah, at this point, it requests blocks from peers via gossip? If so, I assume read/write sets are passed around right? In this scenario, there is no validation done? Just directly apply read sets?
berserkr (Mon, 14 Aug 2017 23:42:48 GMT):
@yacovm when a node fails and needs to use gossip to catch up, it will see the latest block size to be blah, at this point, it requests blocks from peers via gossip? If so, I assume read/write sets are passed around right? In this scenario, there is no validation done? Just directly apply write sets?
yacovm (Tue, 15 Aug 2017 05:01:50 GMT):
@berserkr there is valodation
SotirisAlfonsos (Wed, 16 Aug 2017 08:47:20 GMT):
Hello. I am blocking the network connection on both peers in an org so they do not get the new blocks. When i reconnect, the leader gets the blocks but he does not gossip them to the second peer. ( The second peer closed the endpoint connection to the leader while they were both offline ). How can i re-establish the gossip connection? I am still in beta so could this just be an old issue?
yacovm (Wed, 16 Aug 2017 09:24:18 GMT):
Hmmm anything interesting im the logs?
SotirisAlfonsos (Wed, 16 Aug 2017 09:27:29 GMT):
```2017-08-16 08:11:16.595 UTC [gossip/discovery] getDeadMembers -> WARN 07b Haven't heard from p��ޜ<
\�ڒ�\�e�\��}�����e�`�({ for 26.075709713s
2017-08-16 08:11:16.595 UTC [gossip/discovery] expireDeadMembers -> WARN 07c Entering [[112 209 212 222 156 60 11 92 214 218 146 161 92 140 101 159 92 185 209 125 192 231 136 0 209 241 101 170 96 160 40 123]]
2017-08-16 08:11:16.595 UTC [gossip/discovery] expireDeadMembers -> WARN 07d Closing connection to Endpoint: peer1.org1.example.com:7056, InternalEndpoint: peer1.org1.example.com:7056, PKI-ID: [112 209 212 222 156 60 11 92 214 218 146 161 92 140 101 159 92 185 209 125 192 231 136 0 209 241 101 170 96 160 40 123], Metadata: []
2017-08-16 08:11:16.595 UTC [gossip/discovery] expireDeadMembers -> WARN 07e Exiting
2017-08-16 08:17:08.261 UTC [blocksProvider] DeliverBlocks -> WARN 07f [mychannel] Receive error: Attempts (1) or elapsed time (6m44.286605248s) exhausted
2017-08-16 08:17:09.215 UTC [blocksProvider] DeliverBlocks -> WARN 080 [mychannel2] Receive error: Attempts (1) or elapsed time (6m44.901977168s) exhausted
```
SotirisAlfonsos (Wed, 16 Aug 2017 09:27:29 GMT):
```2017-08-16 08:11:16.595 UTC [gossip/discovery] getDeadMembers -> WARN 07b Haven't heard from p��ޜ<
\�ڒ�\�e�\��}�����e�`�({ for 26.075709713s
2017-08-16 08:11:16.595 UTC [gossip/discovery] expireDeadMembers -> WARN 07c Entering [[112 209 212 222 156 60 11 92 214 218 146 161 92 140 101 159 92 185 209 125 192 231 136 0 209 241 101 170 96 160 40 123]]
2017-08-16 08:11:16.595 UTC [gossip/discovery] expireDeadMembers -> WARN 07d Closing connection to Endpoint: peer1.org1.example.com:7056, InternalEndpoint: peer1.org1.example.com:7056, PKI-ID: [112 209 212 222 156 60 11 92 214 218 146 161 92 140 101 159 92 185 209 125 192 231 136 0 209 241 101 170 96 160 40 123], Metadata: []
2017-08-16 08:11:16.595 UTC [gossip/discovery] expireDeadMembers -> WARN 07e Exiting
2017-08-16 08:17:08.261 UTC [blocksProvider] DeliverBlocks -> WARN 07f [mychannel] Receive error: Attempts (1) or elapsed time (6m44.286605248s) exhausted
2017-08-16 08:17:09.215 UTC [blocksProvider] DeliverBlocks -> WARN 080 [mychannel2] Receive error: Attempts (1) or elapsed time (6m44.901977168s) exhausted
```
That is right when i disconnected both peers. When they are back they are not gossiping anymore. I send a transaction which they both sign but again non leader does not receive the gossip about the block
SotirisAlfonsos (Wed, 16 Aug 2017 09:27:29 GMT):
```2017-08-16 08:11:16.595 UTC [gossip/discovery] getDeadMembers -> WARN 07b Haven't heard from p��ޜ<
\�ڒ�\�e�\��}�����e�`�({ for 26.075709713s
2017-08-16 08:11:16.595 UTC [gossip/discovery] expireDeadMembers -> WARN 07c Entering [[112 209 212 222 156 60 11 92 214 218 146 161 92 140 101 159 92 185 209 125 192 231 136 0 209 241 101 170 96 160 40 123]]
2017-08-16 08:11:16.595 UTC [gossip/discovery] expireDeadMembers -> WARN 07d Closing connection to Endpoint: peer1.org1.example.com:7056, InternalEndpoint: peer1.org1.example.com:7056, PKI-ID: [112 209 212 222 156 60 11 92 214 218 146 161 92 140 101 159 92 185 209 125 192 231 136 0 209 241 101 170 96 160 40 123], Metadata: []
2017-08-16 08:11:16.595 UTC [gossip/discovery] expireDeadMembers -> WARN 07e Exiting
2017-08-16 08:17:08.261 UTC [blocksProvider] DeliverBlocks -> WARN 07f [mychannel] Receive error: Attempts (1) or elapsed time (6m44.286605248s) exhausted
2017-08-16 08:17:09.215 UTC [blocksProvider] DeliverBlocks -> WARN 080 [mychannel2] Receive error: Attempts (1) or elapsed time (6m44.901977168s) exhausted
```
That is right when i disconnected both peers. When they are back they are not gossiping anymore. I send a transaction which they both sign but again the non leader does not receive the gossip about the block
yacovm (Wed, 16 Aug 2017 12:03:41 GMT):
so @SotirisAlfonsos
yacovm (Wed, 16 Aug 2017 12:03:56 GMT):
you're saying that you blocked connection between peers?
yacovm (Wed, 16 Aug 2017 12:04:05 GMT):
and they don't reconnect?
yacovm (Wed, 16 Aug 2017 12:04:14 GMT):
they are supposed to re-connect periodically
SotirisAlfonsos (Wed, 16 Aug 2017 12:09:34 GMT):
They did not reconnect. Even when i invoked and queried transactions on them. By the way, when they disconnected, they both closed the endpoint with each other. In case you recall, was there a change on that from beta to rc?
yacovm (Wed, 16 Aug 2017 12:10:04 GMT):
how much time passed?
yacovm (Wed, 16 Aug 2017 12:10:15 GMT):
between the time you disconnected
yacovm (Wed, 16 Aug 2017 12:10:20 GMT):
until the time you re-connected?
yacovm (Wed, 16 Aug 2017 12:10:29 GMT):
> By the way, when they disconnected, they both closed the endpoint with each other.
Makes sense
SotirisAlfonsos (Wed, 16 Aug 2017 12:15:28 GMT):
10 minutes give or take
yacovm (Wed, 16 Aug 2017 12:20:37 GMT):
hmmm
yacovm (Wed, 16 Aug 2017 12:20:44 GMT):
why are you using beta though? :(
yacovm (Wed, 16 Aug 2017 12:20:56 GMT):
can't you upgrade to release?
SotirisAlfonsos (Wed, 16 Aug 2017 12:32:41 GMT):
Did not have the time to do so. Will try to sync next week and will get back to you.
yacovm (Wed, 16 Aug 2017 12:36:54 GMT):
ok cool thanks
kelvinzhong (Thu, 17 Aug 2017 14:56:56 GMT):
Has joined the channel.
kelvinzhong (Thu, 17 Aug 2017 14:59:09 GMT):
@yacovm we also cross this problem
yacovm (Thu, 17 Aug 2017 14:59:33 GMT):
on which version?
kelvinzhong (Thu, 17 Aug 2017 14:59:43 GMT):
1.0
kelvinzhong (Thu, 17 Aug 2017 15:00:04 GMT):
we got three peers distribute in different PC
kelvinzhong (Thu, 17 Aug 2017 15:00:52 GMT):
likes few hours, it would have a chance some peers disconnect with the others and never reconnect
yacovm (Thu, 17 Aug 2017 15:01:16 GMT):
hmmm really?
yacovm (Thu, 17 Aug 2017 15:01:27 GMT):
what do they print when they disconnect?
yacovm (Thu, 17 Aug 2017 15:02:30 GMT):
why do they disconnect though?
kelvinzhong (Thu, 17 Aug 2017 15:02:51 GMT):
i haven't look into the logs, but it appears so, after one day, we execute another transaction, one of the peer didn't receive the transaction, the block height is less than others
yacovm (Thu, 17 Aug 2017 15:03:22 GMT):
interesting... so you basically keep the peers online and after a day they disconnect?
kelvinzhong (Thu, 17 Aug 2017 15:03:35 GMT):
unless we restart the peer, it would not synchronize with the others
yacovm (Thu, 17 Aug 2017 15:03:42 GMT):
I see
yacovm (Thu, 17 Aug 2017 15:03:47 GMT):
I'll check it out
yacovm (Thu, 17 Aug 2017 15:03:50 GMT):
thanks
yacovm (Thu, 17 Aug 2017 15:04:00 GMT):
but if you can show me logs
yacovm (Thu, 17 Aug 2017 15:04:02 GMT):
that'd help
kelvinzhong (Thu, 17 Aug 2017 15:04:34 GMT):
seems the peers on the same computer would not happen
yacovm (Thu, 17 Aug 2017 15:05:07 GMT):
if the peers are in the same computer it doesn't happen
yacovm (Thu, 17 Aug 2017 15:05:11 GMT):
but in different computers it does?
kelvinzhong (Thu, 17 Aug 2017 15:05:38 GMT):
i'm not sure, since our environment now all in different computers
yacovm (Thu, 17 Aug 2017 15:05:53 GMT):
ok, any chance you get me some logs to look at ?
kelvinzhong (Thu, 17 Aug 2017 15:06:29 GMT):
but when i test on the same computer haven't appear this error
kelvinzhong (Thu, 17 Aug 2017 15:07:18 GMT):
Message Attachments
kelvinzhong (Thu, 17 Aug 2017 15:07:27 GMT):
i'm not sure if this is the right logs for this issue
yacovm (Thu, 17 Aug 2017 15:08:21 GMT):
no no
yacovm (Thu, 17 Aug 2017 15:08:24 GMT):
i need gossip logs
kelvinzhong (Thu, 17 Aug 2017 15:09:00 GMT):
i don't have now...let me check tomorrow
yacovm (Thu, 17 Aug 2017 15:09:07 GMT):
ok cool
yacovm (Thu, 17 Aug 2017 15:09:09 GMT):
thanks
kelvinzhong (Thu, 17 Aug 2017 15:09:09 GMT):
here is 11PM now
yacovm (Thu, 17 Aug 2017 15:09:15 GMT):
understood
kelvinzhong (Thu, 17 Aug 2017 15:10:13 GMT):
btw, the eventhub also would disconnect sometimes, where should i ask
kelvinzhong (Thu, 17 Aug 2017 15:10:13 GMT):
btw, the eventhub would also disconnect sometimes, where should i ask
yacovm (Thu, 17 Aug 2017 15:10:20 GMT):
in #fabric-peer-endorser-committer
kelvinzhong (Thu, 17 Aug 2017 15:10:26 GMT):
thanks
yacovm (Thu, 17 Aug 2017 15:10:31 GMT):
sure
Eman0 (Thu, 17 Aug 2017 20:24:22 GMT):
Has joined the channel.
Eman0 (Thu, 17 Aug 2017 21:07:22 GMT):
I know that the genesis block contains the anchor peers addresses. But when one of them sends “alive” message, how do the others know that this message is from that anchor peer? is it just from the source IP?
yacovm (Thu, 17 Aug 2017 21:57:19 GMT):
why do they need to know that the message is from an anchor peer?
yacovm (Thu, 17 Aug 2017 21:57:30 GMT):
it doesn't matter which peer the message is from
Eman0 (Thu, 17 Aug 2017 23:35:47 GMT):
Actually, I just want to know how peers learn the addresses of each other's. So when a peer receives an "alive" message, how does it know the address of the sender?
Eman0 (Fri, 18 Aug 2017 05:47:39 GMT):
Actually, I just want to know how peers learn the addresses of each other's
yacovm (Fri, 18 Aug 2017 06:53:01 GMT):
each peer gossips about its own address to other peers
CarlXK (Fri, 18 Aug 2017 08:36:32 GMT):
Has joined the channel.
Eman0 (Fri, 18 Aug 2017 13:31:07 GMT):
Is that in the "alive" message ? And is the purpose of anchor peers just to be discovered by newly connected peers ?
yacovm (Fri, 18 Aug 2017 13:47:36 GMT):
Yep
yacovm (Sat, 19 Aug 2017 10:31:43 GMT):
@kelvinzhong @SotirisAlfonsos I tried reproducing your problem with a modified peer that has a 1 Minute times where it does a re-scan and expiration of certificates (instead of the default hours and day ) and it doesn't reproduce :(
yacovm (Sat, 19 Aug 2017 10:32:07 GMT):
Could it be that it's something with the network in your environment?
sampath06 (Sat, 19 Aug 2017 11:22:16 GMT):
Has joined the channel.
kelvinzhong (Mon, 21 Aug 2017 03:30:57 GMT):
@yacovm sadly, we just reproduce this problem...one of the peer disconnect with the others, the block high is less than the others. but we forget to open the debug logs......so we have to wait for the problem occurs next time:joy:
kelvinzhong (Mon, 21 Aug 2017 03:30:57 GMT):
@yacovm sadly, we just reproduce this problem...one of the peer disconnect with the others, the block height is less than the others. but we forget to open the debug logs......so we have to wait for the problem occurs next time:joy:
kelvinzhong (Mon, 21 Aug 2017 03:30:57 GMT):
@yacovm sadly, we just reproduce this problem...one of the peer disconnect with the others, the block height is less than the others. but we forgot to open the debug logs......so we have to wait for the problem occurs next time:joy:
yacovm (Mon, 21 Aug 2017 05:47:32 GMT):
You dont
yacovm (Mon, 21 Aug 2017 05:47:50 GMT):
You can change debug level dynamically while the leer is running
yacovm (Mon, 21 Aug 2017 05:48:03 GMT):
But that might br too late...
yacovm (Mon, 21 Aug 2017 05:48:03 GMT):
But that might be too late...
kelvinzhong (Mon, 21 Aug 2017 05:54:13 GMT):
:joy: yes, we will wait....
kelvinzhong (Mon, 21 Aug 2017 06:25:31 GMT):
@yacovm BTW, how to change the debug level dynamically...we used to change the docker-compose file setting....
yacovm (Mon, 21 Aug 2017 06:26:49 GMT):
Wait actually i'm not sure that still works
kelvinzhong (Mon, 21 Aug 2017 06:34:30 GMT):
well...if it still working please let me know:grin:
kelvinzhong (Mon, 21 Aug 2017 06:40:48 GMT):
one more question:grin: the setting "- CORE_PEER_GOSSIP_ORGLEADER=true", it's stand for the leader peer that orderer would send the transaction first, or it's stand for the anchor peer that would communicate across the organization?
kelvinzhong (Mon, 21 Aug 2017 06:48:17 GMT):
i saw the example from fabric-java-sdk, all peers set as CORE_PEER_GOSSIP_EXTERNALENDPOINT, but there are comment line of CORE_PEER_GOSSIP_BOOTSTRAP and CORE_PEER_GOSSIP_ORGLEADER, not sure what would be the usage of those, could you share how to choose the setting of those?
yacovm (Mon, 21 Aug 2017 06:49:59 GMT):
`./build/bin/peer logging setlevel gossip DEBUG`
yacovm (Mon, 21 Aug 2017 06:50:04 GMT):
@kelvinzhong ^
yacovm (Mon, 21 Aug 2017 06:50:44 GMT):
So the org leader setting makes gossip connect to the ordering service
yacovm (Mon, 21 Aug 2017 06:51:05 GMT):
there is another setting that makes gossip discuss who will connect to the orderer on behalf of the other peers
yacovm (Mon, 21 Aug 2017 06:51:18 GMT):
and then some peer is elected and it is the one that connects to the orderer
yacovm (Mon, 21 Aug 2017 06:51:21 GMT):
per channel, of course
kelvinzhong (Mon, 21 Aug 2017 06:54:02 GMT):
thanks for the explaination!
yacovm (Mon, 21 Aug 2017 06:54:51 GMT):
`CORE_PEER_GOSSIP_EXTERNALENDPOINT` - makes your peer evailable to external organizations
yacovm (Mon, 21 Aug 2017 06:56:08 GMT):
`CORE_PEER_GOSSIP_BOOTSTRAP` - a peer or a few peers from your org that gossip connects to them upon startup to establish membership
yacovm (Mon, 21 Aug 2017 06:56:13 GMT):
@kelvinzhong ^
kelvinzhong (Mon, 21 Aug 2017 06:56:43 GMT):
cool!
kelvinzhong (Mon, 21 Aug 2017 06:58:55 GMT):
but the anchor peer seems has been set in the genesis block, why would it need to be set in the compose file
kelvinzhong (Mon, 21 Aug 2017 06:59:44 GMT):
and the leader peer could be chosen automatically right?
yacovm (Mon, 21 Aug 2017 06:59:44 GMT):
bootstrap
yacovm (Mon, 21 Aug 2017 06:59:45 GMT):
not anchor
yacovm (Mon, 21 Aug 2017 06:59:49 GMT):
they are different
kelvinzhong (Mon, 21 Aug 2017 06:59:56 GMT):
CORE_PEER_GOSSIP_EXTERNALENDPOINT` - makes your peer evailable to external organizations
kelvinzhong (Mon, 21 Aug 2017 07:00:00 GMT):
i mean this one
yacovm (Mon, 21 Aug 2017 07:00:01 GMT):
> and the leader peer could be chosen automatically right?
if leader election is on, yes
yacovm (Mon, 21 Aug 2017 07:00:12 GMT):
this has nothing to do with anchor peers
kelvinzhong (Mon, 21 Aug 2017 07:01:31 GMT):
well... as i read from the doc says the anchor peer is used to communicate with different organization, is that correct?
yacovm (Mon, 21 Aug 2017 07:02:28 GMT):
yes and no. It is used to know about peers from other organizations
yacovm (Mon, 21 Aug 2017 07:02:37 GMT):
but once the peers know each other they don't need the anchor peer anymore
yacovm (Mon, 21 Aug 2017 07:02:44 GMT):
unless they are shut down, that is
kelvinzhong (Mon, 21 Aug 2017 07:02:55 GMT):
that's new....
yacovm (Mon, 21 Aug 2017 07:03:00 GMT):
That's not new
kelvinzhong (Mon, 21 Aug 2017 07:03:04 GMT):
thanks very much for this
kelvinzhong (Mon, 21 Aug 2017 07:03:29 GMT):
i haven't seen this from the doc
yacovm (Mon, 21 Aug 2017 07:04:57 GMT):
what doc
kelvinzhong (Mon, 21 Aug 2017 07:05:13 GMT):
http://hyperledger-fabric.readthedocs.io/en/latest/
kelvinzhong (Mon, 21 Aug 2017 07:08:12 GMT):
once there has the design doc for the fabric.... i probably learn from that
vdods (Wed, 23 Aug 2017 23:10:04 GMT):
Has joined the channel.
scottz (Thu, 24 Aug 2017 21:54:49 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=t3Z6GGW6k3yQ4Nv8Q) @yacovm Could we provide a list of peers to be bootstrap peers in core.yaml, or using docker-compose?
```
scottz (Thu, 24 Aug 2017 21:54:49 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=t3Z6GGW6k3yQ4Nv8Q) @yacovm Are you saying we could somehow provide a LIST of peers to be bootstrap peers in core.yaml?
```
peer:
gossip:
bootstrap: 127.0.0.1:7051
```
And can we specify more than one using docker-compose when using config variable CORE_PEER_GOSSIP_BOOTSTRAP? Or are you implying we could simply specify a different bootstrap peer when we configure each peer?
```
peer0.org1.example.com:
environment:
- CORE_PEER_GOSSIP_BOOTSTRAP=peer1.org1.example.com:7051
peer1.org1.example.com:
environment:
- CORE_PEER_GOSSIP_BOOTSTRAP=peer0.org1.example.com:7051
```
nnao (Fri, 25 Aug 2017 17:57:59 GMT):
Has left the channel.
yacovm (Fri, 25 Aug 2017 19:28:24 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=AqGTogz7qJEjSpcNL)
yacovm (Fri, 25 Aug 2017 19:29:09 GMT):
> Are you saying we could somehow provide a LIST of peers to be bootstrap peers in core.yaml?
Yes.
> And can we specify more than one using docker-compose when using config variable CORE_PEER_GOSSIP_BOOTSTRAP?
Should work.
@scottz
rsherwood (Thu, 31 Aug 2017 14:13:35 GMT):
Has joined the channel.
rsherwood (Thu, 31 Aug 2017 14:24:07 GMT):
In our network we expect to have only one peer per organization. If a peer has to recover, for example by catching up with missing blocks, where will the peer request blocks from ? Will it go to peers of other organizations on the channel or will it go to one of the orderers for redelivery ? If its peers of other organizations where would it get the address other peer from ?
yacovm (Thu, 31 Aug 2017 14:35:41 GMT):
> Will it go to peers of other organizations on the channel or will it go to one of the orderers for redelivery
Both if it's configured to use leader election
> If its peers of other organizations where would it get the address other peer from ?
From the anchor peers of the channel
rsherwood (Thu, 31 Aug 2017 15:01:52 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=SNJ5ytwiyNHzJRh5Z) @yacovm How do I know if a peer is configured to use leader election? In the config transcation I've seen I've not notice the anchor peer being configured. Is this a mandatory field for an MSP ? If these are not specified does it not go to the other organization ?
yacovm (Thu, 31 Aug 2017 15:08:18 GMT):
CORE_PEER_GOSSIP_LEADER_ELECTION=true
rsherwood (Thu, 31 Aug 2017 16:10:40 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=KARQp86aDMNFhHyKK) @yacovm Sorry I'm what familiar with this setting what does this do? If the system goes to other member peers or the order is there any priority as to which happens ?
daijianw (Fri, 01 Sep 2017 03:57:08 GMT):
Has joined the channel.
VictoriaH (Fri, 08 Sep 2017 15:22:25 GMT):
Has joined the channel.
kostas (Fri, 08 Sep 2017 20:18:00 GMT):
Has left the channel.
qizhang (Tue, 12 Sep 2017 15:52:28 GMT):
Can anyone tell me why the following warnings occur? Thanks!
qizhang (Tue, 12 Sep 2017 15:52:28 GMT):
Can anyone tell me why the following warnings occur? I have a Fabric network with 1 org and 2 peers (peer0 and peer1). The client is sending request to both peers, but this warnings ONLY occur in peer0. Thanks!
qizhang (Tue, 12 Sep 2017 15:52:28 GMT):
Can anyone tell me why the following warnings occur? I have a Fabric network with 1 org and 2 peers (peer0 and peer1). The client is sending request to both peers, but these warnings ONLY occur in peer0. Thanks!
qizhang (Tue, 12 Sep 2017 15:52:28 GMT):
Can anyone tell me why the following warnings occur? I have a Fabric network with 1 org and 2 peers (peer0 and peer1). The client is sending requests to both peers, but these warnings ONLY occur in peer0. Thanks!
qizhang (Tue, 12 Sep 2017 15:52:43 GMT):
`peer0.org1.example.com | 2017-09-12 15:46:56.522 UTC [gossip/state] handleStateResponse -> WARN 2311 Payload with sequence number 4657 was received earlier
peer0.org1.example.com | 2017-09-12 15:46:56.523 UTC [gossip/state] handleStateResponse -> WARN 2312 Payload with sequence number 4658 was received earlier
peer0.org1.example.com | 2017-09-12 15:46:56.524 UTC [gossip/state] handleStateResponse -> WARN 2313 Payload with sequence number 4659 was received earlier
peer0.org1.example.com | 2017-09-12 15:46:56.525 UTC [gossip/state] handleStateResponse -> WARN 2314 Payload with sequence number 4660 was received earlier
peer0.org1.example.com | 2017-09-12 15:46:56.526 UTC [gossip/state] handleStateResponse -> WARN 2315 Payload with sequence number 4661 was received earlier
peer0.org1.example.com | 2017-09-12 15:46:56.527 UTC [gossip/state] handleStateResponse -> WARN 2316 Payload with sequence number 4662 was received earlier
peer0.org1.example.com | 2017-09-12 15:46:56.528 UTC [gossip/state] handleStateResponse -> WARN 2317 Payload with sequence number 4663 was received earlier
peer0.org1.example.com | 2017-09-12 15:46:56.528 UTC [gossip/state] handleStateResponse -> WARN 2318 Payload with sequence number 4664 was received earlier
peer0.org1.example.com | 2017-09-12 15:46:56.529 UTC [gossip/state] handleStateResponse -> WARN 2319 Payload with sequence number 4665 was received earlier
peer0.org1.example.com | 2017-09-12 15:46:56.530 UTC [gossip/state] handleStateResponse -> WARN 231a Payload with sequence number 4666 was received earlier`
qizhang (Tue, 12 Sep 2017 15:52:43 GMT):
```peer0.org1.example.com | 2017-09-12 15:46:56.522 UTC [gossip/state] handleStateResponse -> WARN 2311 Payload with sequence number 4657 was received earlier
peer0.org1.example.com | 2017-09-12 15:46:56.523 UTC [gossip/state] handleStateResponse -> WARN 2312 Payload with sequence number 4658 was received earlier
peer0.org1.example.com | 2017-09-12 15:46:56.524 UTC [gossip/state] handleStateResponse -> WARN 2313 Payload with sequence number 4659 was received earlier
peer0.org1.example.com | 2017-09-12 15:46:56.525 UTC [gossip/state] handleStateResponse -> WARN 2314 Payload with sequence number 4660 was received earlier
peer0.org1.example.com | 2017-09-12 15:46:56.526 UTC [gossip/state] handleStateResponse -> WARN 2315 Payload with sequence number 4661 was received earlier
peer0.org1.example.com | 2017-09-12 15:46:56.527 UTC [gossip/state] handleStateResponse -> WARN 2316 Payload with sequence number 4662 was received earlier
peer0.org1.example.com | 2017-09-12 15:46:56.528 UTC [gossip/state] handleStateResponse -> WARN 2317 Payload with sequence number 4663 was received earlier
peer0.org1.example.com | 2017-09-12 15:46:56.528 UTC [gossip/state] handleStateResponse -> WARN 2318 Payload with sequence number 4664 was received earlier
peer0.org1.example.com | 2017-09-12 15:46:56.529 UTC [gossip/state] handleStateResponse -> WARN 2319 Payload with sequence number 4665 was received earlier
peer0.org1.example.com | 2017-09-12 15:46:56.530 UTC [gossip/state] handleStateResponse -> WARN 231a Payload with sequence number 4666 was received earlier```
yacovm (Tue, 12 Sep 2017 17:07:59 GMT):
@qizhang I think it's nothing to worry about, it happens when you get a block from the orderer or from some peer while trying to do state transfer
yacovm (Tue, 12 Sep 2017 17:08:33 GMT):
( @C0rWin correct me if I'm wrong)
yacovm (Tue, 12 Sep 2017 17:08:59 GMT):
You might want to try to turn the logging verbosity level to debug
yacovm (Tue, 12 Sep 2017 17:09:10 GMT):
and see what messages you receive besides it
qizhang (Tue, 12 Sep 2017 17:14:30 GMT):
thanks @yacovm
qizhang (Tue, 12 Sep 2017 17:15:08 GMT):
do you have any thought on why these warnings only occur in peer0 but not peer1? @yacovm
qizhang (Tue, 12 Sep 2017 17:15:58 GMT):
The message was actually from the function 'antiEntropy()', and this function is supposed to run on each peer, right?
qizhang (Tue, 12 Sep 2017 17:15:58 GMT):
The message was actually from the function `antiEntropy()`, and this function is supposed to run on each peer, right?
yacovm (Tue, 12 Sep 2017 17:17:41 GMT):
> do you have any thought on why these warnings only occur in peer0 but not peer1? @yacovm
My bet would be that peer0 doesn't connect to the ordering service, and peer1 does.
yacovm (Tue, 12 Sep 2017 17:17:43 GMT):
You can verify it from logs...
yacovm (Tue, 12 Sep 2017 17:18:18 GMT):
> The message was actually from the function `antiEntropy()`, and this function is supposed to run on each peer, right?
Well yeah, but there is a place in the code where it checks if it's needed to fetch blocks from other peers... if you're the most up to date peer, you don't fetch blocks from peers
qizhang (Tue, 12 Sep 2017 17:19:01 GMT):
I see. Can you let me know how to verify from the logs that whether a peer has connecte to the orderer?
qizhang (Tue, 12 Sep 2017 17:19:55 GMT):
Also, even if peer0 does not connect to the orderer, why it receives blocks that it already has?
qizhang (Tue, 12 Sep 2017 17:19:55 GMT):
Also, even if peer0 does not connect to the orderer, why it keep receiving blocks that it already has?
qizhang (Tue, 12 Sep 2017 17:19:55 GMT):
Also, even if peer0 does not connect to the orderer, why it keeps receiving blocks that it already has?
yacovm (Tue, 12 Sep 2017 17:20:50 GMT):
yeah of course
yacovm (Tue, 12 Sep 2017 17:20:53 GMT):
it prints it....
yacovm (Tue, 12 Sep 2017 17:21:23 GMT):
> Also, even if peer0 does not connect to the orderer, why it keeps receiving blocks that it already has?
because it might fetch N blocks at a time, because it batches them
yacovm (Tue, 12 Sep 2017 17:21:31 GMT):
but it might already received them from a peer
qizhang (Tue, 12 Sep 2017 17:23:37 GMT):
Normally, a peer is supposed to receive blocks from the orderer, right?
qizhang (Tue, 12 Sep 2017 17:25:10 GMT):
If it does not connect to the orderer, the peer then needs to ask its neighbor peers for the blocks. Then, if it receives blocks from its neighbors, why it still receives those blocks from the orderer?
CallMain (Thu, 14 Sep 2017 07:12:09 GMT):
Has joined the channel.
yuanlv (Fri, 15 Sep 2017 21:51:02 GMT):
Has joined the channel.
vdods (Sun, 17 Sep 2017 18:59:49 GMT):
Has left the channel.
yyyyyyy9 (Wed, 20 Sep 2017 01:43:59 GMT):
Has joined the channel.
Colonel_HLE (Wed, 20 Sep 2017 12:20:12 GMT):
Has joined the channel.
thirdstage (Thu, 21 Sep 2017 05:45:24 GMT):
Has joined the channel.
HYOSEOPKWON (Thu, 21 Sep 2017 05:51:16 GMT):
Has joined the channel.
SimonOberzan (Thu, 21 Sep 2017 08:17:23 GMT):
Has joined the channel.
SimonOberzan (Thu, 21 Sep 2017 10:52:41 GMT):
Hi. I had a network with 2 organizations with 2 peers each. I have added another peer to Org2, and peer0.org2 (gossip bootstrap and anchor peer) & peer1.org2 seem fine but peers peer0.org1 and peer1.org1 keep giving following errors: ```[gossip/comm] func1 -> WARN 490 peer2.org2.example.com:7051, PKIid:[72 126 38 46 227 64 227 123 167 189 213 77 113 92 32 12 16 209 206 157 93 166 4 30 240 80 13 109 231 102 174 58] isn't responsive: EOF
[gossip/discovery] expireDeadMembers -> WARN 491 Entering [[72 126 38 46 227 64 227 123 167 189 213 77 113 92 32 12 16 209 206 157 93 166 4 30 240 80 13 109 231 102 174 58]]
[gossip/discovery] expireDeadMembers -> WARN 492 Closing connection to Endpoint: peer2.org2.example.com:7051, InternalEndpoint: , PKI-ID: [72 126 38 46 227 64 227 123 167 189 213 77 113 92 32 12 16 209 206 157 93 166 4 30 240 80 13 109 231 102 174 58], Metadata: []
[gossip/discovery] expireDeadMembers -> WARN 493 Exiting ```
SimonOberzan (Thu, 21 Sep 2017 10:52:41 GMT):
Hi. I had a network with 2 organizations with 2 peers each. I have added another peer to Org2, and peer0.org2 (gossip bootstrap and anchor peer) & peer1.org2 seem fine but peers peer0.org1 and peer1.org1 keep giving following errors: ```[gossip/comm] func1 -> WARN 490 peer2.org2.example.com:7051, PKIid:[72 126 38 46 227 64 227 123 167 189 213 77 113 92 32 12 16 209 206 157 93 166 4 30 240 80 13 109 231 102 174 58] isn't responsive: EOF
[gossip/discovery] expireDeadMembers -> WARN 491 Entering [[72 126 38 46 227 64 227 123 167 189 213 77 113 92 32 12 16 209 206 157 93 166 4 30 240 80 13 109 231 102 174 58]]
[gossip/discovery] expireDeadMembers -> WARN 492 Closing connection to Endpoint: peer2.org2.example.com:7051, InternalEndpoint: , PKI-ID: [72 126 38 46 227 64 227 123 167 189 213 77 113 92 32 12 16 209 206 157 93 166 4 30 240 80 13 109 231 102 174 58], Metadata: []
[gossip/discovery] expireDeadMembers -> WARN 493 Exiting ```
How can I make peer of Org1 "accept" the new peer? Or should I disable the external gossip endpoint, and what would be the advantage/disadvantage?
SimonOberzan (Thu, 21 Sep 2017 10:52:41 GMT):
Hi. I had a network with 2 organizations with 2 peers each. I have added another peer to Org2, and peer0.org2 (gossip bootstrap and anchor peer) & peer1.org2 seem fine but peers peer0.org1 and peer1.org1 keep giving following errors: ```[gossip/comm] func1 -> WARN 490 peer2.org2.example.com:7051, PKIid:[72 126 38 46 227 64 227 123 167 189 213 77 113 92 32 12 16 209 206 157 93 166 4 30 240 80 13 109 231 102 174 58] isn't responsive: EOF
[gossip/discovery] expireDeadMembers -> WARN 491 Entering [[72 126 38 46 227 64 227 123 167 189 213 77 113 92 32 12 16 209 206 157 93 166 4 30 240 80 13 109 231 102 174 58]]
[gossip/discovery] expireDeadMembers -> WARN 492 Closing connection to Endpoint: peer2.org2.example.com:7051, InternalEndpoint: , PKI-ID: [72 126 38 46 227 64 227 123 167 189 213 77 113 92 32 12 16 209 206 157 93 166 4 30 240 80 13 109 231 102 174 58], Metadata: []
[gossip/discovery] expireDeadMembers -> WARN 493 Exiting ```
Logs of the new peer:
```[gossip/comm] authenticateRemotePeer -> WARN 1c8 Identity store rejected 172.20.0.3:43412 : Peer Identity [0a 07 4f...] cannot be validated. No MSP found able to do that.
[gossip/comm] GossipStream -> ERRO 1c9 Authentication failed: Peer Identity [0a 07 4f...] cannot be validated. No MSP found able to do that.
[gossip/gossip] handleMessage -> WARN 1cc Message GossipMessage: tag:EMPTY alive_msg:
SimonOberzan (Thu, 21 Sep 2017 10:52:41 GMT):
Hi. I had a network with 2 organizations with 2 peers each. I have added another peer to Org2, and peer0.org2 (gossip bootstrap and anchor peer) & peer1.org2 seem fine but peers peer0.org1 and peer1.org1 keep giving following errors: ```[gossip/comm] func1 -> WARN 490 peer2.org2.example.com:7051, PKIid:[72 126 38 46 227 64 227 123 167 189 213 77 113 92 32 12 16 209 206 157 93 166 4 30 240 80 13 109 231 102 174 58] isn't responsive: EOF
[gossip/discovery] expireDeadMembers -> WARN 491 Entering [[72 126 38 46 227 64 227 123 167 189 213 77 113 92 32 12 16 209 206 157 93 166 4 30 240 80 13 109 231 102 174 58]]
[gossip/discovery] expireDeadMembers -> WARN 492 Closing connection to Endpoint: peer2.org2.example.com:7051, InternalEndpoint: , PKI-ID: [72 126 38 46 227 64 227 123 167 189 213 77 113 92 32 12 16 209 206 157 93 166 4 30 240 80 13 109 231 102 174 58], Metadata: []
[gossip/discovery] expireDeadMembers -> WARN 493 Exiting ```
Logs of the new peer:
```[gossip/comm] authenticateRemotePeer -> WARN 1c8 Identity store rejected 172.20.0.3:43412 : Peer Identity [0a 07 4f...] cannot be validated. No MSP found able to do that.
[gossip/comm] GossipStream -> ERRO 1c9 Authentication failed: Peer Identity [0a 07 4f...] cannot be validated. No MSP found able to do that.
[gossip/gossip] handleMessage -> WARN 1cc Message GossipMessage: tag:EMPTY alive_msg:
SimonOberzan (Thu, 21 Sep 2017 10:52:41 GMT):
Hi. I had a network with 2 organizations with 2 peers each. I have added another peer to Org2, and peer0.org2 (gossip bootstrap and anchor peer) & peer1.org2 seem fine but peers peer0.org1 and peer1.org1 keep giving following errors: ```[gossip/comm] func1 -> WARN 490 peer2.org2.example.com:7051, PKIid:[72 126 38 46 227 64 227 123 167 189 213 77 113 92 32 12 16 209 206 157 93 166 4 30 240 80 13 109 231 102 174 58] isn't responsive: EOF
[gossip/discovery] expireDeadMembers -> WARN 491 Entering [[72 126 38 46 227 64 227 123 167 189 213 77 113 92 32 12 16 209 206 157 93 166 4 30 240 80 13 109 231 102 174 58]]
[gossip/discovery] expireDeadMembers -> WARN 492 Closing connection to Endpoint: peer2.org2.example.com:7051, InternalEndpoint: , PKI-ID: [72 126 38 46 227 64 227 123 167 189 213 77 113 92 32 12 16 209 206 157 93 166 4 30 240 80 13 109 231 102 174 58], Metadata: []
[gossip/discovery] expireDeadMembers -> WARN 493 Exiting ```
Logs of the new peer:
```[gossip/comm] authenticateRemotePeer -> WARN 1c8 Identity store rejected 172.20.0.3:43412 : Peer Identity [0a 07 4f...] cannot be validated. No MSP found able to do that.
[gossip/comm] GossipStream -> ERRO 1c9 Authentication failed: Peer Identity [0a 07 4f...] cannot be validated. No MSP found able to do that.
[gossip/gossip] handleMessage -> WARN 1cc Message GossipMessage: tag:EMPTY alive_msg:
yacovm (Thu, 21 Sep 2017 11:35:57 GMT):
@SimonOberzan it looks like the MSP of the new peer isnt configuree correctly
yacovm (Thu, 21 Sep 2017 11:35:57 GMT):
@SimonOberzan it looks like the MSP of the new peer isnt configured correctly
SimonOberzan (Thu, 21 Sep 2017 11:43:34 GMT):
@yacovm Do you have any suggestion on how I can check what is wrong? Those are my environmental variables on the new peer:
```- CORE_PEER_ID=peer2.org2.example.com
- CORE_PEER_ADDRESS=peer2.org2.example.com:7051
- CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer2.org2.example.com:7051
- CORE_PEER_DISCOVERY_ROOTNODE=peer0.org2.example.com:7051
- CORE_PEER_CHAINCODELISTENADDRESS=peer2.org2.example.com:7052
- CORE_PEER_GOSSIP_BOOTSTRAP=peer0.org2.example.com:7051
- CORE_PEER_LOCALMSPID=Org2MSP
- CORE_PEER_ADDRESSAUTODETECT=true
SimonOberzan (Thu, 21 Sep 2017 11:43:34 GMT):
@yacovm Do you have any suggestion on how I can check what is wrong? Those are my environmental variables on the new peer:
```- CORE_PEER_ID=peer2.org2.example.com
- CORE_PEER_ADDRESS=peer2.org2.example.com:7051
- CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer2.org2.example.com:7051
- CORE_PEER_DISCOVERY_ROOTNODE=peer0.org2.example.com:7051
- CORE_PEER_CHAINCODELISTENADDRESS=peer2.org2.example.com:7052
- CORE_PEER_GOSSIP_BOOTSTRAP=peer0.org2.example.com:7051
- CORE_PEER_LOCALMSPID=Org2MSP
- CORE_PEER_ADDRESSAUTODETECT=true
- CORE_PEER_GOSSIP_USELEADERELECTION=true
SimonOberzan (Thu, 21 Sep 2017 11:43:34 GMT):
@yacovm Do you have any suggestion on how I can check what is wrong? Those are my environmental variables on the new peer:
```- CORE_PEER_ID=peer2.org2.example.com
- CORE_PEER_ADDRESS=peer2.org2.example.com:7051
- CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer2.org2.example.com:7051
- CORE_PEER_DISCOVERY_ROOTNODE=peer0.org2.example.com:7051
- CORE_PEER_CHAINCODELISTENADDRESS=peer2.org2.example.com:7052
- CORE_PEER_GOSSIP_BOOTSTRAP=peer0.org2.example.com:7051
- CORE_PEER_LOCALMSPID=Org2MSP
- CORE_PEER_ADDRESSAUTODETECT=true
- CORE_PEER_GOSSIP_USELEADERELECTION=true
- CORE_PEER_GOSSIP_ORGLEADER=false
SimonOberzan (Thu, 21 Sep 2017 11:43:34 GMT):
@yacovm Do you have any suggestion on how I can check what is wrong? Those are my environmental variables on the new peer:
```- CORE_PEER_ID=peer2.org2.example.com
- CORE_PEER_ADDRESS=peer2.org2.example.com:7051
- CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer2.org2.example.com:7051
- CORE_PEER_DISCOVERY_ROOTNODE=peer0.org2.example.com:7051
- CORE_PEER_CHAINCODELISTENADDRESS=peer2.org2.example.com:7052
- CORE_PEER_GOSSIP_BOOTSTRAP=peer0.org2.example.com:7051
- CORE_PEER_LOCALMSPID=Org2MSP
- CORE_PEER_ADDRESSAUTODETECT=true
- CORE_PEER_GOSSIP_USELEADERELECTION=true
- CORE_PEER_GOSSIP_ORGLEADER=false
- CORE_PEER_PROFILE_ENABLED=false
- CORE_PEER_TLS_ENABLED=false
SimonOberzan (Thu, 21 Sep 2017 11:43:34 GMT):
@yacovm Do you have any suggestion on how I can check what is wrong? Those are my environmental variables on the new peer:
```- CORE_PEER_ID=peer2.org2.example.com
- CORE_PEER_ADDRESS=peer2.org2.example.com:7051
- CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer2.org2.example.com:7051
- CORE_PEER_DISCOVERY_ROOTNODE=peer0.org2.example.com:7051
- CORE_PEER_CHAINCODELISTENADDRESS=peer2.org2.example.com:7052
- CORE_PEER_GOSSIP_BOOTSTRAP=peer0.org2.example.com:7051
- CORE_PEER_LOCALMSPID=Org2MSP
- CORE_PEER_ADDRESSAUTODETECT=true
- CORE_PEER_GOSSIP_USELEADERELECTION=true
- CORE_PEER_GOSSIP_ORGLEADER=false
- CORE_PEER_PROFILE_ENABLED=false
- CORE_PEER_TLS_ENABLED=false
- CORE_VM_ENDPOINT=172.16.67.239:2375
- CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=cello_net_solo
SimonOberzan (Thu, 21 Sep 2017 11:43:34 GMT):
@yacovm Do you have any suggestion on how I can check what is wrong? Those are my environmental variables on the new peer:
```- CORE_PEER_ID=peer2.org2.example.com
- CORE_PEER_ADDRESS=peer2.org2.example.com:7051
- CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer2.org2.example.com:7051
- CORE_PEER_DISCOVERY_ROOTNODE=peer0.org2.example.com:7051
- CORE_PEER_CHAINCODELISTENADDRESS=peer2.org2.example.com:7052
- CORE_PEER_GOSSIP_BOOTSTRAP=peer0.org2.example.com:7051
- CORE_PEER_LOCALMSPID=Org2MSP
- CORE_PEER_ADDRESSAUTODETECT=true
- CORE_PEER_GOSSIP_USELEADERELECTION=true
- CORE_PEER_GOSSIP_ORGLEADER=false
- CORE_PEER_PROFILE_ENABLED=false
- CORE_PEER_TLS_ENABLED=false
- CORE_VM_ENDPOINT=172.16.67.239:2375
- CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=cello_net_solo```
I registered and erolled the peer on CA2 and used generated msp.
yacovm (Thu, 21 Sep 2017 11:48:49 GMT):
What about the msp id?
SimonOberzan (Thu, 21 Sep 2017 11:51:17 GMT):
- CORE_PEER_LOCALMSPID=Org2MSP? Is ther any other msp id?
yacovm (Thu, 21 Sep 2017 12:22:51 GMT):
Thats the one
qizhang (Fri, 22 Sep 2017 04:47:20 GMT):
Anyone can explain what does the following function do? The comment does not seem to be clear to me. Thanks!
qizhang (Fri, 22 Sep 2017 04:47:44 GMT):
``` // Deliver in order messages into the incoming channel
go s.deliverPayloads()```
C0rWin (Fri, 22 Sep 2017 06:59:34 GMT):
@qizhang blocks are gossiped hence received at arbitrary order, this method leverages buffer which takes care to commit blocks to the ledger according to the natural order
qizhang (Fri, 22 Sep 2017 16:11:50 GMT):
@C0rWin I know that each peer has a buffer to store the received blocks, does this method receives the gossiped blocks and put them into the buffer, so that the committing routine can commit the blocks from the buffer to the ledger, Or this method takes the block from the buffer and then calls the committing routine to commit to the ledger?
C0rWin (Fri, 22 Sep 2017 18:52:39 GMT):
@qizhang this method polls buffer and commits the next in oder block once available
qizhang (Fri, 22 Sep 2017 19:34:09 GMT):
@C0rWin You mentioned that the blocks come to the buffer in an arbitrary order, and I saw that this method just pops each block out of the buffer and then commit. Does it mean that this method commit the block in the same order that the blocks arrive in the buffer, which is also arbitrary?
C0rWin (Sat, 23 Sep 2017 08:07:59 GMT):
@qizhang buffer takes care to re-order blocks in natural order, hence blocks are committed in the original order according to sequence numbers generated by orderer
avi-nyc (Sun, 24 Sep 2017 00:03:36 GMT):
Has joined the channel.
yushan (Tue, 26 Sep 2017 09:16:57 GMT):
Has joined the channel.
qizhang (Fri, 29 Sep 2017 17:59:49 GMT):
If the client does not register any event, does the Fabric server still generate events but just not send them?
yacovm (Fri, 29 Sep 2017 18:04:38 GMT):
it's just like pub/sub works
yacovm (Fri, 29 Sep 2017 18:04:44 GMT):
and the answer is yes
yacovm (Fri, 29 Sep 2017 18:04:49 GMT):
but this isn't related to gossip
qizhang (Fri, 29 Sep 2017 18:10:19 GMT):
Thanks @yacovm . Yeah, but I was not sure which channel shall I ask, please advise.
yacovm (Fri, 29 Sep 2017 18:10:29 GMT):
#fabric-peer-endorser-committer
qizhang (Fri, 29 Sep 2017 18:10:57 GMT):
OK, I am going to join
CodeReaper (Mon, 02 Oct 2017 15:03:08 GMT):
Has joined the channel.
CodeReaper (Mon, 02 Oct 2017 15:05:40 GMT):
Hey, I was wondering if I start a byfn network and the start a peer separately and assign that peer's gossip external endpoint to the core peer, Will that peer update all the blocks into its own ledger by itself??
CodeReaper (Mon, 02 Oct 2017 15:06:00 GMT):
Or do i require some process beforehand??
yacovm (Mon, 02 Oct 2017 15:09:41 GMT):
what is a core peer?
CodeReaper (Mon, 02 Oct 2017 15:41:04 GMT):
Anchor peer? is'nt that the one mentioned in the compose file?? Will it sync itself with any peer it has its core_peer_address set to for that matter?
yacovm (Mon, 02 Oct 2017 15:46:05 GMT):
I don't understand what you're asking, sorry.
yacovm (Mon, 02 Oct 2017 15:46:07 GMT):
can you rephrase?
CodeReaper (Mon, 02 Oct 2017 15:49:17 GMT):
I have a docker network running, supposedly of byfn network. Now If i start a separate peer setting all the defaults of any other peer in the network. will the peer sync all the blocks in its own ledger by itself?
yacovm (Mon, 02 Oct 2017 15:50:38 GMT):
ah so the peer needs:
1) To join a channel
2) To know some anchor peer. I'm not sure if byfn has anchor peers in the configtx.yaml
yacovm (Mon, 02 Oct 2017 15:50:53 GMT):
but, *very important - do not forget!*
You have to give your peer a new certificate.
yacovm (Mon, 02 Oct 2017 15:50:59 GMT):
do not give it a certificate of an existing peer
CodeReaper (Mon, 02 Oct 2017 15:51:48 GMT):
Yes this was gonna be the second question, but i was wondering if the syncing process requires a certificate.
yacovm (Mon, 02 Oct 2017 15:53:15 GMT):
Fabric is a blockchain for a permissioned network
yacovm (Mon, 02 Oct 2017 15:53:24 GMT):
everything you do that interacts with another network node
yacovm (Mon, 02 Oct 2017 15:53:28 GMT):
requires some authentication ;)
yacovm (Mon, 02 Oct 2017 15:53:51 GMT):
so, in gossip's case we have a check in the code that sees if someone uses the same certificate as some other peer
yacovm (Mon, 02 Oct 2017 15:55:49 GMT):
now, what will happen is that both peers won't be able to "see" each other
yacovm (Mon, 02 Oct 2017 15:56:06 GMT):
they'll just complain that there is some peer with the same certificate, but with a different endpoint than their own
CodeReaper (Mon, 02 Oct 2017 15:56:23 GMT):
Got it, thanks.
CodeReaper (Mon, 02 Oct 2017 15:56:47 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=wi2fWCodW6qYWGT3J) How can i achieve this
CodeReaper (Mon, 02 Oct 2017 15:57:21 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=XWec7EmzP92dZ36Xc) How can i achieve this
CodeReaper (Mon, 02 Oct 2017 15:57:56 GMT):
register and enroll the peer from inside the peer itself??
CodeReaper (Mon, 02 Oct 2017 15:58:05 GMT):
Not sure how that would work
CodeReaper (Mon, 02 Oct 2017 15:58:52 GMT):
The peer doesnt have direct communication with the CA I think
CodeReaper (Mon, 02 Oct 2017 15:59:50 GMT):
Do I register from a peer whose already enrolled, sounds right to me
yacovm (Mon, 02 Oct 2017 16:27:22 GMT):
not from within the peer
yacovm (Mon, 02 Oct 2017 16:27:28 GMT):
externally with a commandline
CodeReaper (Mon, 02 Oct 2017 16:40:21 GMT):
sorry for the rapid fire @yacovm , I have just 2 more queries.
1)You said a peer needs to join a channel. does that mean it will it sync blocks of that particular channel only? That would disrupt the chain right? So maybe it will just commit changes from other channels also, updating its world state and the blockchain.
CodeReaper (Mon, 02 Oct 2017 16:42:19 GMT):
2)"To know some anchor peer". Will it be the peer the new peer should set its core_peer_gossip_address to?? How exactly will it know a peer.
AlekNS (Wed, 04 Oct 2017 05:14:22 GMT):
Has joined the channel.
yacovm (Fri, 06 Oct 2017 15:30:43 GMT):
@CodeReaper dont know why or how i forgot about these pending questions :slight_smile: . I guess i'm overwhelmed with questions these days. So w.r.t 1)yes. Why would it distrupt the chain? Syncing blocks is a read. 2) the anxhor peer(s) are just endpoints of known prers, encoded in configuration blocks.
Asara (Fri, 06 Oct 2017 20:43:21 GMT):
Has joined the channel.
yoheiueda (Wed, 11 Oct 2017 05:27:53 GMT):
Has joined the channel.
tennenjl (Wed, 11 Oct 2017 14:39:41 GMT):
Has joined the channel.
baramustafa (Wed, 11 Oct 2017 22:27:58 GMT):
Has joined the channel.
asaningmaxchain (Thu, 12 Oct 2017 00:33:32 GMT):
Has joined the channel.
william123 (Thu, 12 Oct 2017 13:41:23 GMT):
Has left the channel.
minollo (Thu, 12 Oct 2017 14:58:32 GMT):
Has joined the channel.
minollo (Thu, 12 Oct 2017 15:05:17 GMT):
Hi @yacovm ; I have been exploring the concept of leader peers in the past few days; sorry if my questions are naive or basic - I have been looking for answers elsewhere, but I couldn't find them. From what I understand, in order to actually have a single leader peer elected in the context of a channel, for a single org, I need to explicitly configure gossip.bootstrap before I start the peers and join them to a channel; that's required to make sure each peer node knows which other peer nodes it needs to interact with in the context of gossip - I believe. Is there any possibility (or any future plan) to make it possible for peers to "discover" each other through the ordering service or other means? Relying on the configuration files settings for a leader peer to be elected (a setting which is not even required) seems to be quite error prone - and I suppose it also forces peer nodes to be bounced if I want a new peer to be recognized in the election process.
minollo (Thu, 12 Oct 2017 15:05:17 GMT):
Hi @yacovm ; I have been exploring the concept of leader peers in the past few days; sorry if my questions are naive or basic - I have been looking for answers elsewhere, but I couldn't find them. From what I understand, in order to actually have a single leader peer elected in the context of a channel, for a single org, I need to explicitly configure gossip.bootstrap before I start the peers and join them to a channel; that's required to make sure each peer node knows which other peer nodes it needs to interact with in the context of gossip - I believe. Is there any possibility (or any future plan) to make it possible for peers to "discover" each other through the ordering service or other means? Relying on the configuration files settings for a leader peer to be elected (a setting which is not even required) seems to be quite error prone - and I suppose it also forces peer nodes to be bounced if I want a new peer to be recognized in the election process.
In general, I'm trying to understand if I should rely on the concept of leader peer, to possibly assigning some specific roles to the leader peer moving forward; right now, the fact that leader peers seem to be an "optional" entity, surprises me a little.
yacovm (Thu, 12 Oct 2017 15:18:32 GMT):
@minollo well - you can also use anchor peers
yacovm (Thu, 12 Oct 2017 15:18:38 GMT):
you don't have to use bootstrap peers
yacovm (Thu, 12 Oct 2017 15:18:48 GMT):
if your org has an anchor peer you don't need a bootstrap peer.
toddinpal (Thu, 12 Oct 2017 15:29:35 GMT):
@yacovm perhaps I'm missing something, but I can't find a good description of the role of anchor and leader peers and how they operate. They're barely mentioned at all in the hyperledger architecture section. Is there a good description somewhere of what they do and how they do it?
yacovm (Thu, 12 Oct 2017 15:30:01 GMT):
I can just explain it here
toddinpal (Thu, 12 Oct 2017 15:30:46 GMT):
yeah, but I suspect others have similar questions... Seems like a good section for the architecture section of the documentation
minollo (Thu, 12 Oct 2017 15:35:41 GMT):
I agree with @toddinpal that an explanation of the architecture in the documentation would help; but in the meanwhile, @yacovm , I would be happy to hear here about the role of anchor and leader peers, and for what purposes I can/should rely on one or the other. My initial understanding of anchor peers (and the reason why I didn't pay much attention to them) is that they can be >1 for a channel/org - while I was looking for a peer (what I was assuming should be the leader peer) to have a unique role in the context of a channel/org...
yacovm (Thu, 12 Oct 2017 16:07:01 GMT):
an anchor peer is an endpoint of a peer from an organization
that is useful for multi-organizational channels
in which the orgs can find the peers of other organizations using the anchor peers
i.e org0 and org1 each have an anchor peer: peer0.org0 and peer0.org1
so the update tx is sent to the channel to all organizations
and with that - peer0.org1 can know about peer0.org0 and can tell peer1.org1, peer2.org1, etc. about peer0.org0 but also about peer1.org0, peer2.org0
because peer0.org0 knows about peer1.org0, and peer2.org0 and would tell other organizations about them
yacovm (Thu, 12 Oct 2017 16:07:11 GMT):
a leader peer is the peer (in an org) that connects to the orderer
yacovm (Thu, 12 Oct 2017 16:07:22 GMT):
on behalf of its organization, for the channel
yacovm (Thu, 12 Oct 2017 16:07:29 GMT):
a leader peer is per channel and for an org
minollo (Thu, 12 Oct 2017 16:18:05 GMT):
@yacovm , thanks for the explanation; luckily enough, that fits the understanding I had. And I am apparently correct in focusing on leader peers, as I was looking for that one peer with a special role in the context of one channel for an org. Going back to my original question(s), it looks like the notion of the leader peer is not "mandated", but optional - and entirely driven by configuration files. Is that ever going to change (so that I can count on the fact that I will *always* have a single leader peer per channel, per org? Is that always going to be driven by the gossip configuration information?
yacovm (Thu, 12 Oct 2017 16:18:41 GMT):
it's not strictly via config files
yacovm (Thu, 12 Oct 2017 16:18:52 GMT):
a leader can be statically set via the config
yacovm (Thu, 12 Oct 2017 16:19:00 GMT):
or it can be dynamically elected by its peers
yacovm (Thu, 12 Oct 2017 16:19:03 GMT):
(pun itended)
yacovm (Thu, 12 Oct 2017 16:19:03 GMT):
(pun intended)
yacovm (Thu, 12 Oct 2017 16:19:30 GMT):
> so that I can count on the fact that I will *always* have a single leader peer per channel, per org
No one promises you that
yacovm (Thu, 12 Oct 2017 16:19:52 GMT):
why do you have such a need?
minollo (Thu, 12 Oct 2017 16:25:09 GMT):
I was looking for a peer who would a role of possibly storing information in some peculiar way - either because I might want to replicate at least one peer's ledger in a slow but resilient storage, or because I might want to store state or history in a persistence storage which allows me to run richer queries than what couchdb state queries or leveldb history iterations allow me to do. And that would need to be per channel, per org - which fits quite well with the way a leader peer is defined. Makes sense?
BTW, one more question specific to leader peers and gossip protocol: assuming that gossip is configured to have peers elect a leader, I assume there is a mechanism to make sure that if the current leader is no more responsive a new one is elected; is that assumption correct? If yes, how long would it typically take for the new leader to be elected after the old one dies?
yacovm (Thu, 12 Oct 2017 16:27:02 GMT):
I understand your odd need, but I'm not sure how you're going to accomplish that - store in the state in a storage that isn't couch or levelDB. are you going to change the fabric code?
yacovm (Thu, 12 Oct 2017 16:27:24 GMT):
> I assume there is a mechanism to make sure that if the current leader is no more responsive a new one is elected; is that assumption correct?
Of course.
yacovm (Thu, 12 Oct 2017 16:28:27 GMT):
> If yes, how long would it typically take for the new leader to be elected after the old one dies?
hmm it's configurable:
```
election:
# Longest time peer waits for stable membership during leader election startup (unit: second)
startupGracePeriod: 15s
# Interval gossip membership samples to check its stability (unit: second)
membershipSampleInterval: 1s
# Time passes since last declaration message before peer decides to perform leader election (unit: second)
leaderAliveThreshold: 10s
# Time between peer sends propose message and declares itself as a leader (sends declaration message) (unit: second)
leaderElectionDuration: 5s
yacovm (Thu, 12 Oct 2017 16:28:32 GMT):
the above are the defaults
minollo (Thu, 12 Oct 2017 16:29:12 GMT):
Excellent, that helps a lot, thanks!
jeffgarratt (Thu, 12 Oct 2017 17:46:10 GMT):
Has joined the channel.
jyellick (Thu, 12 Oct 2017 17:46:17 GMT):
Has joined the channel.
jeffgarratt (Thu, 12 Oct 2017 17:47:21 GMT):
@yacovm @jyellick getting this error from non-upgraded peers when upgrading from 1.0.3 -> master branch
jeffgarratt (Thu, 12 Oct 2017 17:47:24 GMT):
```2017-10-12 17:44:41.310 UTC [gossip/comm] createConnection -> WARN 3cf Remote endpoint claims to be a different peer, expected [124 221 79 7 0 236 182 116 79 110 214 197 15 194 2 53 151 130 178 225 166 85 26 121 93 13 105 124 95 241 23
1 155] but got [132 147 215 211 28 42 217 236 228 175 192 198 194 101 124 238 159 102 125 255 140 192 66 95 245 223 6 118 143 186 157 39]
yacovm (Thu, 12 Oct 2017 17:48:51 GMT):
hmmm... let me ask you something
yacovm (Thu, 12 Oct 2017 17:49:04 GMT):
how much time pass between the upgrades?
yacovm (Thu, 12 Oct 2017 17:49:23 GMT):
did you upgrade one of the peers?
yacovm (Thu, 12 Oct 2017 17:49:31 GMT):
and the other peer complains about this?
yacovm (Thu, 12 Oct 2017 17:50:31 GMT):
So the scenario is- you upgrade one of the peers and the other one complains?
yacovm (Thu, 12 Oct 2017 17:56:41 GMT):
Basically, this may happen if you replace a peer's identity and then restart it.
The peer that tries to reconnect to the peer that was restarted assumes it is a certain peer and it sees a different peer on the other side
yacovm (Thu, 12 Oct 2017 17:56:53 GMT):
therefore, it suspects it, and aborts the connection.
yacovm (Thu, 12 Oct 2017 17:57:09 GMT):
It's none fatal, unless it's an anchor peer or a bootstrap peer.
yacovm (Thu, 12 Oct 2017 17:57:52 GMT):
Generally, what you can do is just wait for a while until the peer is considered "really dead" by the peer that re-connects
yacovm (Thu, 12 Oct 2017 17:58:01 GMT):
so it will not try to re-connect to that ip:port anymore
jyellick (Thu, 12 Oct 2017 17:58:26 GMT):
Upgrade some of the peers
jyellick (Thu, 12 Oct 2017 17:58:41 GMT):
There was no identity replacement though, only upgrading the binary
yacovm (Thu, 12 Oct 2017 17:59:06 GMT):
ah but the master branch changed the `Serialize()` method
yacovm (Thu, 12 Oct 2017 17:59:06 GMT):
ah but the master branch changed the `mspIdentity.Serialize()` method
yacovm (Thu, 12 Oct 2017 17:59:16 GMT):
so to gossip it is in a way a change of the identity ;)
jyellick (Thu, 12 Oct 2017 18:05:45 GMT):
So... what's the fix? Does it just work after a while?
yacovm (Thu, 12 Oct 2017 18:05:52 GMT):
It should.
yacovm (Thu, 12 Oct 2017 18:06:03 GMT):
you can do something else though
yacovm (Thu, 12 Oct 2017 18:06:09 GMT):
when you launch the upgraded peer
yacovm (Thu, 12 Oct 2017 18:06:20 GMT):
if it's in the same org, for example - give it the none-upgrade peer
yacovm (Thu, 12 Oct 2017 18:06:25 GMT):
as a bootstrap peer
yacovm (Thu, 12 Oct 2017 18:06:32 GMT):
then it'll connect to it
yacovm (Thu, 12 Oct 2017 18:06:36 GMT):
instead the other way around
yacovm (Thu, 12 Oct 2017 18:06:48 GMT):
and since it's a fresh PKI-ID, it'll be OK
jyellick (Thu, 12 Oct 2017 18:07:03 GMT):
In the bdd, each peer points to the other
jyellick (Thu, 12 Oct 2017 18:07:21 GMT):
How long is the timeout before things should 'just work'?
yacovm (Thu, 12 Oct 2017 18:07:24 GMT):
wdym to the other? like `--> --> -->` in a circle?
jyellick (Thu, 12 Oct 2017 18:07:26 GMT):
We also tried restarting the peer binary
yacovm (Thu, 12 Oct 2017 18:07:37 GMT):
if you restart both of them it should work 100%
yacovm (Thu, 12 Oct 2017 18:07:41 GMT):
since the memory is wiped out
jyellick (Thu, 12 Oct 2017 18:07:44 GMT):
PeerA points to PeerB points to PeerA, yes
yacovm (Thu, 12 Oct 2017 18:07:59 GMT):
did you try restarting both?
jeffgarratt (Thu, 12 Oct 2017 18:08:03 GMT):
not yet
yacovm (Thu, 12 Oct 2017 18:08:04 GMT):
just to check the claim
jeffgarratt (Thu, 12 Oct 2017 18:08:05 GMT):
can give it a go
jyellick (Thu, 12 Oct 2017 18:08:16 GMT):
Not at the same time
yacovm (Thu, 12 Oct 2017 18:08:47 GMT):
hmmm wait a second
yacovm (Thu, 12 Oct 2017 18:09:07 GMT):
if the bootstrap configuration points in a circle
yacovm (Thu, 12 Oct 2017 18:09:14 GMT):
that means they should be connected
yacovm (Thu, 12 Oct 2017 18:09:24 GMT):
the upgraded peer should be connected to the none upgraded
yacovm (Thu, 12 Oct 2017 18:09:34 GMT):
can you perhaps do `tcpdump` or `netstat` and confirm?
yacovm (Thu, 12 Oct 2017 18:10:11 GMT):
or, if the gossip logs are in DEBUG you should see lots of noise
jyellick (Thu, 12 Oct 2017 18:11:06 GMT):
We are still getting the warning after 35 minutes
yacovm (Thu, 12 Oct 2017 18:11:49 GMT):
right but I'm trying to figure out - are they connected?
yacovm (Thu, 12 Oct 2017 18:11:54 GMT):
or not?
jyellick (Thu, 12 Oct 2017 18:11:57 GMT):
We restarted peer3, with an updated binary. Then restarted peer2 with the existing binary because of the warning
jyellick (Thu, 12 Oct 2017 18:12:13 GMT):
I don't know if they are connected, but the spamming of that warning would seem to imply to me that they are not?
yacovm (Thu, 12 Oct 2017 18:12:36 GMT):
who are the peers in the story? just peer3 and peer2?
jyellick (Thu, 12 Oct 2017 18:12:55 GMT):
Yes, peer2 and peer3 are the only two peers in an org
jyellick (Thu, 12 Oct 2017 18:13:45 GMT):
(peer0 and peer1 are Org1, peer2 and peer3 are Org2, we upgraded peer3 with a binary from master, saw the warning, restarted peer2 with the existing v1.0.3 binary)
jyellick (Thu, 12 Oct 2017 18:14:08 GMT):
peer0 and peer2 are anchor peers
jyellick (Thu, 12 Oct 2017 18:14:40 GMT):
Though external endpoint is not set for anyone, so there should be no cross org gossip
yacovm (Thu, 12 Oct 2017 18:14:54 GMT):
> we upgraded peer3 with a binary from master, saw the warning, restarted peer2 with the existing v1.0.3 binary
hmm?
yacovm (Thu, 12 Oct 2017 18:15:02 GMT):
so you upgraded both?
jyellick (Thu, 12 Oct 2017 18:15:16 GMT):
No, peer2 and peer3 were executing with the v1.0.3 binary
jyellick (Thu, 12 Oct 2017 18:15:25 GMT):
We stopped peer3, upgraded his binary to master, then started him again
jyellick (Thu, 12 Oct 2017 18:15:41 GMT):
We saw the warning, suspected it was a cache problem from peer2, so, simply restarted peer2 (no change in binary)
yacovm (Thu, 12 Oct 2017 18:15:55 GMT):
wait so you restarted both peers?
jyellick (Thu, 12 Oct 2017 18:15:58 GMT):
Yes
jyellick (Thu, 12 Oct 2017 18:16:12 GMT):
Not at the same time, first peer3, then peer2
yacovm (Thu, 12 Oct 2017 18:17:25 GMT):
hmmm let me take a look how much time an identity can even circulate before it's purged
yacovm (Thu, 12 Oct 2017 18:18:35 GMT):
ok so it's an hour. but I still don't understand how can 35 minutes pass and a peer attempts to probe another peer
yacovm (Thu, 12 Oct 2017 18:19:00 GMT):
the fact that it expects a certain PKI-ID says that it got that from a previous AliveMessage
yacovm (Thu, 12 Oct 2017 18:19:23 GMT):
wait a second
yacovm (Thu, 12 Oct 2017 18:19:27 GMT):
BDD is docker
yacovm (Thu, 12 Oct 2017 18:19:33 GMT):
and in docker you get random IP addresses
yacovm (Thu, 12 Oct 2017 18:19:50 GMT):
well, semi random
yacovm (Thu, 12 Oct 2017 18:20:21 GMT):
... I still don't understand how can 35 min pass and still have warnings though
yacovm (Thu, 12 Oct 2017 18:22:37 GMT):
ah wait
yacovm (Thu, 12 Oct 2017 18:23:38 GMT):
ok I think I understand what's going on @jyellick / @jeffgarratt
yacovm (Thu, 12 Oct 2017 18:23:51 GMT):
when a peer connects to an anchor peer or a bootstrap peer it computes its PKI-ID
yacovm (Thu, 12 Oct 2017 18:24:24 GMT):
so we have a v1.0.3 peers that computes the PKI-ID and expects the upgraded peer on the other side to identify itself in a certain way
yacovm (Thu, 12 Oct 2017 18:24:24 GMT):
so we have a v1.0.3 peers that computes the PKI-ID and expects the upgraded peer on the other side to identify itself in a certain way and present its PKI-ID which is computed differently.
yacovm (Thu, 12 Oct 2017 18:24:53 GMT):
but since the computation is different... they will never be able to communicate
yacovm (Thu, 12 Oct 2017 18:24:53 GMT):
but since the computation is different... they will never be able to communicate because the side that opens the connection suspects the other side is not who it expects he is.
jeffgarratt (Thu, 12 Oct 2017 18:25:34 GMT):
I don't see that as a problem
jyellick (Thu, 12 Oct 2017 18:25:42 GMT):
So... gossip will be broken in a mixed v1.0.x v1.1 network forever?
jeffgarratt (Thu, 12 Oct 2017 18:25:44 GMT):
most folks will coordinate the upgrade process I presume
yacovm (Thu, 12 Oct 2017 18:25:49 GMT):
yep!
jeffgarratt (Thu, 12 Oct 2017 18:25:58 GMT):
good nough
yacovm (Thu, 12 Oct 2017 18:26:10 GMT):
I'm sorry, my code is over-paranoid :(
jeffgarratt (Thu, 12 Oct 2017 18:26:25 GMT):
so you would assume once I upgrade both peers, gossip should reestablish?
yacovm (Thu, 12 Oct 2017 18:26:30 GMT):
yep
jeffgarratt (Thu, 12 Oct 2017 18:26:33 GMT):
k.. will verify
jeffgarratt (Thu, 12 Oct 2017 18:26:35 GMT):
thnx!
yacovm (Thu, 12 Oct 2017 18:26:42 GMT):
sure.
yacovm (Thu, 12 Oct 2017 18:28:29 GMT):
@mastersingh24 so looks like we won't get mixed peers gossiping anyway.... ^
yacovm (Thu, 12 Oct 2017 18:31:21 GMT):
@jyellick @jeffgarratt something suddenly strikes me as odd. Did no peer complain about handshake problems?
yacovm (Thu, 12 Oct 2017 18:32:25 GMT):
when they handshake, they exchange PKI-IDs and verify the PKI-IDs match the expected computation...
yacovm (Thu, 12 Oct 2017 18:33:15 GMT):
should be some `c.logger.Warningf("Identity store rejected %s : %v", remoteAddress, err)` message IIUC
jeffgarratt (Thu, 12 Oct 2017 18:34:31 GMT):
do not see that
jeffgarratt (Thu, 12 Oct 2017 18:34:31 GMT):
@yacovm do not see that
yacovm (Thu, 12 Oct 2017 20:06:03 GMT):
@jyellick @jeffgarratt I take back what I said:
> so we have a v1.0.3 peers that computes the PKI-ID and expects the upgraded peer on the other side to identify itself in a certain way and present its PKI-ID which is computed differently.
> but since the computation is different... they will never be able to communicate because the side that opens the connection suspects the other side is not who it expects he is.
I don't think the PKI-ID computation changed between the versions. I'll look into this myself and investigate.
yacovm (Thu, 12 Oct 2017 21:17:50 GMT):
@jyellick , @jeffgarratt I did it locally it everything seems to work for me :/
yacovm (Thu, 12 Oct 2017 21:18:03 GMT):
I started both peers as v1.0.3
yacovm (Thu, 12 Oct 2017 21:18:16 GMT):
and then killed one of them and restarted it as a v1.1
yacovm (Thu, 12 Oct 2017 21:18:22 GMT):
and they gossip just fine
yacovm (Thu, 12 Oct 2017 21:18:55 GMT):
Do you want to help me reproduce this?
yacovm (Thu, 12 Oct 2017 21:19:20 GMT):
only I did it with VMs not with BDD or docker
yacovm (Thu, 12 Oct 2017 22:37:02 GMT):
ok so I'm pretty sure it's from the discovery module trying to re-connect to the dead peer that is PKI-ID was `x` and after the upgrade it changed because the implementation of `Serialize()` changed to `y`
yacovm (Thu, 12 Oct 2017 22:38:10 GMT):
however I'm not sure why it's not purging the message from memory fast enough...
yacovm (Fri, 13 Oct 2017 20:37:53 GMT):
@jyellick , @jeffgarratt @mastersingh24 I investigated this issue, the TLDR is that there is no bug and everything is working as expected:
1) Have 2 peers in v1.0.3
2) Upgrade one of them to v1.1
The v1.0.3 peer now has the new upgraded peer in its alive view, but since https://gerrit.hyperledger.org/r/#/c/13265/17/msp/identities.go@181 was merged, it now sees this peer as a completely different peer because its identity now serialized into new bytes.
Now, it still has the old incarnation of the peer (when it was v1.0.3) in its "dead" view and attempts to re-connect to it.
When it reconnects to it, it does it in 2 stages:
- Probes it via a gRPC RPC which succeeds
- Sends it a membership request to merge its membership view with the peer's view, but gossip is a paranoid communication middleware, and when it establishes connection with new peers it knew before - it expects their PKI-ID to match what it was before (this is so a malicious org won't publish membership with endpoints of an existing org and thus isolate peers from other peers). The communication that re-establishes to the re-incarnated peer performs a handshake (that succeeds) and returns the new PKI-ID (one that it is already connected to, but it's irrelevant) and then it rejects the connection (hence the `createConnection` warning in the log) and complains.
I did an experiment and it seems that after ~ 16 minutes it stops reconnecting because it forgets the peer (by purging it from the dead member list).
This time duration is computed by multiplying `20` by `peer.gossip.aliveExpirationTimeout`.
I tested this on a recompiled v1.0.3 peer on which I added a printf to the part where it deletes the message from the dead list and indeed after that print, there were no new warnings or attempts of reconnections.
yacovm (Fri, 13 Oct 2017 20:37:53 GMT):
@jyellick , @jeffgarratt @mastersingh24 I investigated this issue, the TLDR is that there is no bug and everything is working as expected:
1) Have 2 peers in v1.0.3
2) Upgrade one of them to v1.1
The v1.0.3 peer now has the new upgraded peer in its alive view, but since https://gerrit.hyperledger.org/r/#/c/13265/17/msp/identities.go@181 was merged, it now sees this peer as a completely different peer because its identity now serialized into new bytes.
Now, it still has the old incarnation of the peer (when it was v1.0.3) in its "dead" view and attempts to re-connect to it.
When it reconnects to it, it does it in 2 stages:
- Probes it via a gRPC RPC which succeeds
- Sends it a membership request to merge its membership view with the peer's view, but gossip is a paranoid communication middleware, and when it establishes connection with new peers it knew before - it expects their PKI-ID to match what it was before (this is so a malicious org won't publish membership with endpoints of an existing org and thus isolate peers from other peers). The communication that re-establishes to the re-incarnated peer performs a handshake (that succeeds) and returns the new PKI-ID (one that it is already connected to, but it's irrelevant) and then it rejects the connection because of the mismatch (hence the `createConnection` warning in the log) and complains.
I did an experiment and it seems that after ~ 16 minutes it stops reconnecting because it forgets the peer (by purging it from the dead member list).
This time duration is computed by multiplying `20` by `peer.gossip.aliveExpirationTimeout`.
I tested this on a recompiled v1.0.3 peer on which I added a printf to the part where it deletes the message from the dead list and indeed after that print, there were no new warnings or attempts of reconnections.
yacovm (Fri, 13 Oct 2017 20:37:53 GMT):
@jyellick , @jeffgarratt @mastersingh24 I investigated this issue, the TLDR is that there is no bug and everything is working as expected:
1) Have 2 peers in v1.0.3
2) Upgrade one of them to v1.1
The v1.0.3 peer now has the new upgraded peer in its alive view, but since https://gerrit.hyperledger.org/r/#/c/13265/17/msp/identities.go@181 was merged, it now sees this peer as a completely different peer because its identity now serialized into new bytes.
Now, it still has the old incarnation of the peer (when it was v1.0.3) in its "dead" view and attempts to re-connect to it.
When it reconnects to it, it does it in 2 stages:
- Probes it via a gRPC RPC which succeeds
- Sends it a membership request to merge its membership view with the peer's view, but gossip is a paranoid communication middleware, and when it establishes connection with new peers it knew before - it expects their PKI-ID to match what it was before (this is so a malicious org won't isolate peers from other peers). The communication that re-establishes to the re-incarnated peer performs a handshake (that succeeds) and returns the new PKI-ID (one that it is already connected to, but it's irrelevant) and then it rejects the connection because of the mismatch (hence the `createConnection` warning in the log) and complains.
I did an experiment and it seems that after ~ 16 minutes it stops reconnecting because it forgets the peer (by purging it from the dead member list).
This time duration is computed by multiplying `20` by `peer.gossip.aliveExpirationTimeout`.
I tested this on a recompiled v1.0.3 peer on which I added a printf to the part where it deletes the message from the dead list and indeed after that print, there were no new warnings or attempts of reconnections.
jyellick (Fri, 13 Oct 2017 20:47:57 GMT):
How about the TLDR; what should the admin do on upgrade?
yacovm (Fri, 13 Oct 2017 20:48:15 GMT):
just upgrade and wait for the errors to fade
yacovm (Fri, 13 Oct 2017 20:48:25 GMT):
I tested and they did after 16 min
Asara (Fri, 13 Oct 2017 20:48:48 GMT):
So wait, is it supposed to be a **seamless** upgrade from v1.0.3 to v1.1?
yacovm (Fri, 13 Oct 2017 20:49:19 GMT):
Well gossip is backward compatible
jyellick (Fri, 13 Oct 2017 20:50:19 GMT):
We are striving to allow a mixed-network rolling style upgrade. Where some peers may be upgraded and others may not.
jyellick (Fri, 13 Oct 2017 20:50:54 GMT):
Presently, on restart, the peer changes the way it computes its own PKI-ID which will cause a little log spam from the gossip layer as it tries to sort things out. After 16 minutes, the log spam should go away and things should be fine again.
jyellick (Fri, 13 Oct 2017 20:50:54 GMT):
Presently, on upgrade, the peer changes the way it computes its own PKI-ID which will cause a little log spam from the gossip layer as it tries to sort things out. After 16 minutes, the log spam should go away and things should be fine again.
yacovm (Fri, 13 Oct 2017 20:51:10 GMT):
wait, lets be accurate!
yacovm (Fri, 13 Oct 2017 20:51:26 GMT):
it's not the computation that changes, but the serialized identity
yacovm (Fri, 13 Oct 2017 20:52:26 GMT):
Also... if you have time and mind merging https://gerrit.hyperledger.org/r/#/c/14217/ Jason, it would help :)
It's related to this
jyellick (Fri, 13 Oct 2017 20:52:57 GMT):
> but since https://gerrit.hyperledger.org/r/#/c/13265/17/msp/identities.go@181 was merged
I'm a little confused why that change was snuck into that CR. It seems like, doing this 'right', we should have tied the serialization style to the MSP version added recently by @adc
yacovm (Fri, 13 Oct 2017 20:53:45 GMT):
how would that have helped?
yacovm (Fri, 13 Oct 2017 20:54:09 GMT):
His change set is only good for verifier MSPs
yacovm (Fri, 13 Oct 2017 20:54:20 GMT):
while the identity in the story is produced by the local MSP
yacovm (Fri, 13 Oct 2017 20:54:58 GMT):
also Jason don't forget - gossip is cross channel. There is no notion of "act now as v1.1" in gossip
yacovm (Fri, 13 Oct 2017 20:55:23 GMT):
you can only have 1 identity for your peer at least for now...
jyellick (Fri, 13 Oct 2017 20:57:03 GMT):
14217 seems unmergeable to me as is. This affects the output of `Validate`, no?
yacovm (Fri, 13 Oct 2017 20:57:17 GMT):
it's gossip validate
yacovm (Fri, 13 Oct 2017 20:57:23 GMT):
it's only used in gossip
jyellick (Fri, 13 Oct 2017 20:58:00 GMT):
Oh, I see, I missed that
yacovm (Fri, 13 Oct 2017 20:58:03 GMT):
that's the function that validates identities for gossip... it was written by @adc
yacovm (Fri, 13 Oct 2017 20:58:29 GMT):
we have an interface that is implemented from the MSP building blocks and we use it as a black box so we can mock it for tests
jyellick (Fri, 13 Oct 2017 20:58:36 GMT):
Though it does seem worth arguing that `Valdate` in the MSP in `V1_1` should also invoke this method
yacovm (Fri, 13 Oct 2017 20:59:03 GMT):
> Though it does seem worth arguing that `Valdate` in the MSP in `V1_1` should also invoke this method
I don't understand... can you elaborate?
jyellick (Fri, 13 Oct 2017 20:59:39 GMT):
(This goes back to my general dissatisfaction with the identity interface, that it requires caching and synchronization, because there's no type differentiation based on the checks which have been performed against the identity)
jyellick (Fri, 13 Oct 2017 21:00:18 GMT):
If the PEM field for the type is set incorrectly for an identity, would we not want `Validate` to fail for that identity?
jyellick (Fri, 13 Oct 2017 21:00:18 GMT):
If the PEM field for the type is set incorrectly for an identity, would we not want `msp.Validate` to fail for that identity?
jyellick (Fri, 13 Oct 2017 21:00:18 GMT):
If the PEM field for the type is set incorrectly for an identity, would we not want `SerializedIdentity.Validate` to fail for that identity?
yacovm (Fri, 13 Oct 2017 21:00:48 GMT):
so @adc told me he will follow up on this
yacovm (Fri, 13 Oct 2017 21:01:02 GMT):
but in a different CR and this will be based on the compatibility
yacovm (Fri, 13 Oct 2017 21:01:07 GMT):
otherwise we'll fork.
yacovm (Fri, 13 Oct 2017 21:01:14 GMT):
But this can go in :)
jyellick (Fri, 13 Oct 2017 21:05:47 GMT):
Understood, I was clicking the `->` button and hadn't noticed we had switched out of the `msp` dir and into `gossip`
yacovm (Fri, 13 Oct 2017 21:06:13 GMT):
that's peer/gossip, not gossip :rolling_eyes:
Francis-Guttridge (Mon, 16 Oct 2017 16:01:42 GMT):
Has joined the channel.
rohitadivi (Mon, 16 Oct 2017 21:42:23 GMT):
Has joined the channel.
JosephChang (Wed, 18 Oct 2017 13:09:09 GMT):
Has joined the channel.
thakkarparth007 (Wed, 18 Oct 2017 21:15:41 GMT):
If I want add a peer to multiple channels at a time, do I need to take care that the channel-join requests take place one after another?
Right now, I was adding a peer to 8 channels at once, and got this in peer logs:
```
2017-10-18 21:09:08.978 UTC [gossip/state] directMessage -> WARN 19f Received state transfer request for channel ch5 while expecting channel ch6 skipping request...
2017-10-18 21:09:08.978 UTC [gossip/state] directMessage -> WARN 19e Received state transfer request for channel ch5 while expecting channel ch7 skipping request...
```
This seems to have caused a deadlock - as the peer doesn't join any other channels after joining one of the 8
thakkarparth007 (Wed, 18 Oct 2017 21:15:41 GMT):
If I want add a peer to multiple channels at a time, do I need to take care that the channel-join requests take place one after another?
Right now, I was adding a peer to 8 channels at once, and got this in peer logs:
```
2017-10-18 21:09:08.978 UTC [gossip/state] directMessage -> WARN 19f Received state transfer request for channel ch5 while expecting channel ch6 skipping request...
2017-10-18 21:09:08.978 UTC [gossip/state] directMessage -> WARN 19e Received state transfer request for channel ch5 while expecting channel ch7 skipping request...
```
This seems to have caused a deadlock - as the peer doesn't join any other channels after joining one of the 8
thakkarparth007 (Wed, 18 Oct 2017 21:30:56 GMT):
@yacovm, any idea?
yacovm (Wed, 18 Oct 2017 21:37:34 GMT):
hmm what fabric version is that, first of all?
thakkarparth007 (Wed, 18 Oct 2017 21:41:27 GMT):
1
thakkarparth007 (Wed, 18 Oct 2017 21:41:27 GMT):
v1
thakkarparth007 (Wed, 18 Oct 2017 21:42:46 GMT):
I was going through the code, and doesn't look like it should happen.
```
gossipChan, _ := g.Accept(func(message interface{}) bool {
// Get only data messages
return message.(*proto.GossipMessage).IsDataMsg() &&
bytes.Equal(message.(*proto.GossipMessage).Channel, []byte(chainID))
}, false)
```
in NewGossipStateProvider should filter out messages of different channels, and thus that warning should never be shown
thakkarparth007 (Wed, 18 Oct 2017 21:43:22 GMT):
But that's being shown in directMessage(), not sure how that works.
thakkarparth007 (Wed, 18 Oct 2017 21:45:55 GMT):
https://github.com/hyperledger/fabric/blob/release/gossip/state/state.go#L162
Perhaps here we need to add the chainID equality condition too?
yacovm (Wed, 18 Oct 2017 21:49:18 GMT):
@C0rWin ^
yacovm (Wed, 18 Oct 2017 21:49:25 GMT):
I think he is on to something
C0rWin (Wed, 18 Oct 2017 21:51:05 GMT):
``` connInfo := receivedMsg.GetConnectionInfo()
authErr := mcs.VerifyByChannel(msg.Channel, connInfo.Identity, connInfo.Auth.Signature, connInfo.Auth.SignedData)
if authErr != nil {
logger.Warning("Got unauthorized nodeMetastate transfer request from", string(connInfo.Identity))
return false
}
return true```
C0rWin (Wed, 18 Oct 2017 21:51:22 GMT):
isn't this code should implicitly take care of chanID?
C0rWin (Wed, 18 Oct 2017 21:51:32 GMT):
https://github.com/hyperledger/fabric/blob/release/gossip/state/state.go#L152
C0rWin (Wed, 18 Oct 2017 21:51:52 GMT):
oh
yacovm (Wed, 18 Oct 2017 21:51:54 GMT):
so I think the problem is that the filter just doesn't explicitly filter by channel
yacovm (Wed, 18 Oct 2017 21:52:12 GMT):
but there is still (I think) no real problem, it just spams the logs
C0rWin (Wed, 18 Oct 2017 21:52:15 GMT):
I think because same identity appear in both channels
yacovm (Wed, 18 Oct 2017 21:52:19 GMT):
I _ think _ the logic still should work
C0rWin (Wed, 18 Oct 2017 21:52:22 GMT):
so we need to add an explicit check
yacovm (Wed, 18 Oct 2017 21:52:30 GMT):
yep.
yacovm (Wed, 18 Oct 2017 21:52:35 GMT):
thanks @thakkarparth007
C0rWin (Wed, 18 Oct 2017 21:52:41 GMT):
good catch
thakkarparth007 (Wed, 18 Oct 2017 21:53:21 GMT):
Yay.
thakkarparth007 (Wed, 18 Oct 2017 21:54:13 GMT):
Okay, so can I just add that ```bytes.Equal(message.(*proto.GossipMessage).Channel, []byte(chainID)``` condition somewhere in remoteMsgFilter and it'll work?
thakkarparth007 (Wed, 18 Oct 2017 21:54:37 GMT):
I don't understand what the second argument to Accept does, that's why I'm asking this
thakkarparth007 (Wed, 18 Oct 2017 21:54:37 GMT):
I don't understand what the second argument to g.Accept() does, that's why I'm asking this
yacovm (Wed, 18 Oct 2017 21:57:48 GMT):
yes it should.
yacovm (Wed, 18 Oct 2017 21:57:55 GMT):
the 2nd argument is passthrough
yacovm (Wed, 18 Oct 2017 21:58:09 GMT):
it indicates to the gossip whether it should just return the Accept() of the comm layer
yacovm (Wed, 18 Oct 2017 21:58:16 GMT):
or return the Accept() of its own layer
yacovm (Wed, 18 Oct 2017 21:58:25 GMT):
you want to fix it @thakkarparth007 ?
yacovm (Wed, 18 Oct 2017 21:58:31 GMT):
(upload a fix to gerrit, etc.)
thakkarparth007 (Wed, 18 Oct 2017 22:01:54 GMT):
Okay sure,
thakkarparth007 (Wed, 18 Oct 2017 22:02:06 GMT):
but I'm not familiar with gerrit PRs. I've mostly used github.
yacovm (Wed, 18 Oct 2017 22:02:21 GMT):
oh it's easy - install `git-review`
thakkarparth007 (Wed, 18 Oct 2017 22:03:14 GMT):
Okay, I'll check it out. Do I need to do anything on the JIRA?
yacovm (Wed, 18 Oct 2017 22:05:37 GMT):
yes. you need to open a JIRA and explain the problem, basically
yacovm (Wed, 18 Oct 2017 22:05:47 GMT):
and then when you push the change set,
yacovm (Wed, 18 Oct 2017 22:05:49 GMT):
the title needs to be:
yacovm (Wed, 18 Oct 2017 22:05:58 GMT):
[JIRA-xxx] when xxx is the number of the JIRA
yacovm (Wed, 18 Oct 2017 22:05:58 GMT):
"[JIRA-xxx] bla bla bla "when xxx is the number of the JIRA
yacovm (Wed, 18 Oct 2017 22:06:18 GMT):
I mean - to be prefixed
thakkarparth007 (Wed, 18 Oct 2017 22:07:11 GMT):
Oh okay. I'll get the patch ready, and ping you for details.
qizhang (Thu, 19 Oct 2017 15:20:39 GMT):
Hi everyone, I got the following problem when starting a Fabric network, any thought? Thanks!
qizhang (Thu, 19 Oct 2017 15:20:42 GMT):
```2017-10-18 21:16:20.040 UTC [gossip/comm] authenticateRemotePeer -> WARN 00c Identity store rejected 172.18.0.11:56762 : failed classifying identity: Unable to extract msp.Identity from peer Identity: Peer Identity [0a 08 50 ...... 2d 2d 0a] cannot be validated. No MSP found able to do that.
2017-10-18 21:16:20.040 UTC [gossip/comm] GossipStream -> ERRO 00d Authentication failed: failed classifying identity: Unable to extract msp.Identity from peer Identity: Peer Identity [0a 08 50 ...... 2d 2d 0a] cannot be validated. No MSP found able to do that.```
yacovm (Thu, 19 Oct 2017 15:22:54 GMT):
That's not a gossip problem, that's a configuration problem
yacovm (Thu, 19 Oct 2017 15:22:59 GMT):
the identity isn't valid
yacovm (Thu, 19 Oct 2017 15:23:38 GMT):
it seems to me you forgot to add some peer to a channel
yacovm (Thu, 19 Oct 2017 15:24:05 GMT):
so a peer contacts another peer but their handshake fails because they have no common channel.
They are probably in different organizations
qizhang (Thu, 19 Oct 2017 15:37:27 GMT):
hmm.. will take a look, thanks for the hint @yacovm
sanchezl (Mon, 23 Oct 2017 19:13:21 GMT):
Has joined the channel.
sanchezl (Mon, 23 Oct 2017 19:14:34 GMT):
Orderer - Peer Join-2.png
sanchezl (Mon, 23 Oct 2017 19:15:20 GMT):
@yacovm , @C0rWin Can you please comment on my Peer Join state chart?
qizhang (Mon, 23 Oct 2017 19:36:46 GMT):
Hi everyone, I got following messages from peer log, what could go wrong?
qizhang (Mon, 23 Oct 2017 19:36:58 GMT):
```peer1.org7.example.com | 2017-10-23 19:14:48.555 UTC [gossip/state] queueNewMessage -> WARN 461 Failed adding payload: Ledger height is at 628, cannot enqueue block with sequence of 1289
peer1.org7.example.com | 2017-10-23 19:14:48.555 UTC [gossip/state] queueNewMessage -> WARN 462 Failed adding payload: Ledger height is at 628, cannot enqueue block with sequence of 1288
peer1.org7.example.com | 2017-10-23 19:14:48.555 UTC [gossip/state] queueNewMessage -> WARN 463 Failed adding payload: Ledger height is at 628, cannot enqueue block with sequence of 1287
peer1.org7.example.com | 2017-10-23 19:14:48.556 UTC [gossip/state] queueNewMessage -> WARN 464 Failed adding payload: Ledger height is at 628, cannot enqueue block with sequence of 1290
peer0.org2.example.com | 2017-10-23 19:14:48.607 UTC [gossip/state] queueNewMessage -> WARN 44b Failed adding payload: Ledger height is at 696, cannot enqueue block with sequence of 1302
peer0.org2.example.com | 2017-10-23 19:14:48.609 UTC [gossip/state] queueNewMessage -> WARN 44c Failed adding payload: Ledger height is at 696, cannot enqueue block with sequence of 1304
peer0.org2.example.com | 2017-10-23 19:14:48.609 UTC [gossip/state] queueNewMessage -> WARN 44d Failed adding payload: Ledger height is at 696, cannot enqueue block with sequence of 1303```
yacovm (Mon, 23 Oct 2017 21:40:15 GMT):
@qizhang what fabric version?
yacovm (Mon, 23 Oct 2017 21:41:36 GMT):
@sanchezl I'd say its somewhat accurate but it doesn't really say anything useful apart that peers can be either leaders or not be leaders and can be behind or not. What is the goal of the flow chart?
minollo (Mon, 23 Oct 2017 22:17:48 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=YNZZFdfYMR5iwHujf) @yacovm The one thing the diagram clearly states - which is missing from other descriptions I've seen - is that the recovery procedure is different for leader peers vs. non-leader peers.
yacovm (Mon, 23 Oct 2017 22:18:46 GMT):
well but if leader election is on then a leader can become a non leader and vice versa
minollo (Mon, 23 Oct 2017 22:38:26 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=uJu9BjvHYqJbWt37t) @yacovm Understood; but doesn't that change then the recovery procedure for that peer - if it was a leader, and now it isn't anymore?
sanchezl (Tue, 24 Oct 2017 00:10:27 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=YNZZFdfYMR5iwHujf) @yacovm I'm looking at adding orderer archiving support and the chart is my attempt to understand just what/when the peer needs something from the orderer. In this instance, I see that a non-leader peer does not need anything from the orderer, while a leader peer will need at most, at minimum, the blocks needed to stay up-to-date, at worst, all blocks from block #1.
qizhang (Tue, 24 Oct 2017 15:41:46 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=tMJg84NFYArs7o6CR) I am using the master branch of Fabric, since there are some recent updates included.
yacovm (Tue, 24 Oct 2017 16:29:32 GMT):
@qizhang - does the peer's ledger height advance, or not?
qizhang (Tue, 24 Oct 2017 17:42:17 GMT):
@yacovm yes, it does. All the peers end with the same ledger height
yacovm (Tue, 24 Oct 2017 17:43:04 GMT):
so what is the problem? that they emit warnings?
qizhang (Tue, 24 Oct 2017 17:48:09 GMT):
yes, they just emit warnings, no other problems
qizhang (Tue, 24 Oct 2017 17:48:09 GMT):
yes, they just emit tons of such warnings, no other problems
indira.kalagara (Wed, 25 Oct 2017 12:22:37 GMT):
Has joined the channel.
rishabh1102 (Fri, 27 Oct 2017 12:31:04 GMT):
Has joined the channel.
rishabh1102 (Fri, 27 Oct 2017 12:31:34 GMT):
Hey all, I had a question, what is the best way to minimize gossip time?
rishabh1102 (Fri, 27 Oct 2017 12:32:44 GMT):
I was getting around 400s Gossip time in a 16peer set up, but it suddenly went up to 2.5s in 32peers
rishabh1102 (Fri, 27 Oct 2017 12:32:51 GMT):
400ms*
Vadim (Fri, 27 Oct 2017 12:34:21 GMT):
Has joined the channel.
yacovm (Fri, 27 Oct 2017 13:06:45 GMT):
hmm, 400ms - what are you measuring?
mastersingh24 (Fri, 27 Oct 2017 13:06:46 GMT):
@rishabh1102 - How exactly are you measuring gossip time? And what exactly is your setup - e.g. how many peers per org, etc
mastersingh24 (Fri, 27 Oct 2017 13:06:53 GMT):
@yacovm :)
yacovm (Fri, 27 Oct 2017 13:09:49 GMT):
We made experiments of our own for ~ 16 nodes and they were much less in the latency part. I remember the average number was much lower than 400ms.
For 32 peers you would probably need to crank up the `propagatePeerNum` property
rishabh1102 (Fri, 27 Oct 2017 13:10:59 GMT):
Where is this property set in docker compose? Do you have an example?
yacovm (Fri, 27 Oct 2017 13:11:18 GMT):
since it's x2 the peer number, it makes sense to increase the `propagatePeerNum` at least by 1, but I'd say that 6 peers is a good number, because log(32) is 5 so make it 6 to be certain
yacovm (Fri, 27 Oct 2017 13:11:50 GMT):
ah so the property is not set, therefore it is inherited by core.yaml
yacovm (Fri, 27 Oct 2017 13:11:53 GMT):
which is 3....
yacovm (Fri, 27 Oct 2017 13:12:50 GMT):
also make sure you have at least 4 cores per peer
rishabh1102 (Fri, 27 Oct 2017 13:14:10 GMT):
I have 16 peers and 1 orderer. The block size is 5, so for every 6th successive transaction, I have to wait a certain amount of time before sending it, otherwise it fails
rishabh1102 (Fri, 27 Oct 2017 13:14:19 GMT):
That waiting time is how I'm measuring gossip time
mastersingh24 (Fri, 27 Oct 2017 13:15:26 GMT):
What do you mean "fails"? Are you trying to update the same record?
yacovm (Fri, 27 Oct 2017 13:15:30 GMT):
5 what? inches? KB? apples?
yacovm (Fri, 27 Oct 2017 13:15:30 GMT):
5 what? inches? KB? apples? transactions?
rishabh1102 (Fri, 27 Oct 2017 13:15:40 GMT):
5 transactions
rishabh1102 (Fri, 27 Oct 2017 13:16:12 GMT):
haha
yacovm (Fri, 27 Oct 2017 13:16:32 GMT):
`MaxMessageCount: 5` ?
rishabh1102 (Fri, 27 Oct 2017 13:16:37 GMT):
Yes
mastersingh24 (Fri, 27 Oct 2017 13:16:46 GMT):
And what does fail mean?
yacovm (Fri, 27 Oct 2017 13:17:22 GMT):
right, tha'ts probably MVCC
yacovm (Fri, 27 Oct 2017 13:17:37 GMT):
but do transactions 1,2,3 succeed?
rishabh1102 (Fri, 27 Oct 2017 13:18:46 GMT):
The following Happens:
Cycle 1 -> 1,2,3,4,5 -> All Succeed
Cycle 2-> 1,2,3,3,5 -> 1 fails unless I make my SDK sleep for a certain amount of time, and this repeats on every 1 of the cycle
yacovm (Fri, 27 Oct 2017 13:18:51 GMT):
> That waiting time is how I'm measuring gossip time
This is really not an ideal way to measure gossip. If you want to bench a component of an entire system you should isolate it, for example- measure the time it takes for the leader peer to propagate blocks to the peers.
yacovm (Fri, 27 Oct 2017 13:19:14 GMT):
why does the transaction fail?
yacovm (Fri, 27 Oct 2017 13:19:18 GMT):
is it invalidated by MVCC or what?
rishabh1102 (Fri, 27 Oct 2017 13:19:25 GMT):
By fail, I get 2 types of errors
rishabh1102 (Fri, 27 Oct 2017 13:19:53 GMT):
But, mostly its the following error: Transaction has Failed The proposal responses do not have consistent read write sets
rishabh1102 (Fri, 27 Oct 2017 13:20:42 GMT):
Also, my main aim isn't benchmarking, it is to get the total transaction latency down at the moment
yacovm (Fri, 27 Oct 2017 13:20:48 GMT):
where is that error? the peer or the SDK?
rishabh1102 (Fri, 27 Oct 2017 13:20:52 GMT):
SDK
rishabh1102 (Fri, 27 Oct 2017 13:21:30 GMT):
But I'm assuming that its because not all peers give the same result for the transaction, which would be because not all are updated when I send the new transaction to them
rishabh1102 (Fri, 27 Oct 2017 13:22:26 GMT):
How can I manually set propogatePeerNum?
yacovm (Fri, 27 Oct 2017 13:25:44 GMT):
`CORE_PEER_GOSSIP_PROPAGATEPEERNUM=`
rishabh1102 (Fri, 27 Oct 2017 14:22:06 GMT):
Thanks, I tried changing the variable, but the gossip time is still the same
rishabh1102 (Fri, 27 Oct 2017 14:22:22 GMT):
Is there any other setting which has to be changed along with this?
yacovm (Fri, 27 Oct 2017 15:07:52 GMT):
I told you that running 16 peers in docker on a single physical machine isn't such a good idea...
srongzhe (Sun, 29 Oct 2017 13:28:34 GMT):
Has joined the channel.
Jonny (Mon, 30 Oct 2017 06:45:42 GMT):
Has joined the channel.
KnightTuring (Thu, 02 Nov 2017 16:32:41 GMT):
Has joined the channel.
nickyng (Fri, 10 Nov 2017 03:46:16 GMT):
Has joined the channel.
nickyng (Fri, 10 Nov 2017 04:06:09 GMT):
hi all, i would like to ask whether there is any way to let the gossip leader to retry connecting back to orderer after the connection exhausted?
`2017-11-10 02:29:13.674 UTC [blocksProvider] DeliverBlocks -> WARN 7d3 [channel01] Receive error: Attempts (1) or elapsed time (10m19.967564472s) exhausted`
On my network, when the gossip leader peer is unable to reach the orderer, based on current Fabric (using v1.0.0) deliveryClient reConnectTotalTimeThreshold = 5 minutes, after this threshold, the connection will be exhausted.
When the orderer is resumed reachable, the gossip leader won't retry to connect back even the block provider event is being deregister & register again:
```
2017-11-10 02:39:00.733 UTC [eventhub_producer] validateEventMessage -> DEBU 7d4 ValidateEventMessage starts for signed event 0xc42155d4a0
2017-11-10 02:39:00.737 UTC [eventhub_producer] deRegisterHandler -> DEBU 7d5 deregistering event type: BLOCK
2017-11-10 02:39:00.743 UTC [eventhub_producer] Chat -> DEBU 7d6 Received EOF, ending Chat
2017-11-10 02:39:00.757 UTC [eventhub_producer] validateEventMessage -> DEBU 7d7 ValidateEventMessage starts for signed event 0xc421ce42a0
2017-11-10 02:39:00.759 UTC [eventhub_producer] registerHandler -> DEBU 7d8 registering event type: BLOCK
```
The block providers can only be resumed when restarting the gossip leader, .
```
2017-11-10 03:36:14.846 UTC [deliveryClient] StartDeliverForChannel -> DEBU 362 This peer will pass blocks from orderer service to other peers for channel channel01
2017-11-10 03:36:14.859 UTC [deliveryClient] RequestBlocks -> DEBU 363 Starting deliver with block [33] for channel channel01
```
On one side, the current reConnectTotalTimeThreshold at 5 mins is too idea. There is already setter function but not yet available as a parameter for peer start up configuration.
Is there any other way to let the gossip leader to resume orderer connection for blocks provider purpose?
Thanks.
yacovm (Fri, 10 Nov 2017 07:18:33 GMT):
@nickyng in v1.1 you can customize this 5 minute time
nickyng (Fri, 10 Nov 2017 10:16:47 GMT):
@yacovm thanks. I can see the it will be configurable in v1.1... while we are using v1.0 having app built and tested, and thus, not at a good moment to look into v1.1 which is so-called a preview now...
that's why would like to get advice whether another means to resume it on v1.0 ?? thanks a lot....
yacovm (Fri, 10 Nov 2017 11:21:36 GMT):
Just cherry pick the changr set
yacovm (Fri, 10 Nov 2017 11:21:41 GMT):
And compile the peer
yacovm (Fri, 10 Nov 2017 11:21:45 GMT):
It might work
knagware9 (Fri, 10 Nov 2017 17:29:14 GMT):
Has joined the channel.
rajasekharpippalla (Tue, 14 Nov 2017 05:56:52 GMT):
Has joined the channel.
rajasekharpippalla (Tue, 14 Nov 2017 05:57:01 GMT):
Oracle ---->>> verification module
JPMorgan ---->>> verification module
1. Oracle as well as JPMorgan has done transactions with verfication module.
2. Now verification module can have information for both Oracle and JPMorgan.
3. Oracle should not have access for JPMorgan data and vice-versa.
How can we implement this kind of requirement? Can we implement this by using channels and how we can implement if we have any chances?
C0rWin (Tue, 14 Nov 2017 07:35:12 GMT):
@rajasekharpippalla this channel is for gossip related questions, moved your question into #fabric
shahdhruv 2 (Fri, 17 Nov 2017 19:26:42 GMT):
Has joined the channel.
marksta (Tue, 21 Nov 2017 09:03:31 GMT):
Has joined the channel.
shahdhruv 2 (Wed, 22 Nov 2017 09:07:45 GMT):
Does Hyperledger fabric has its own implementation of Gossip protocol or some library is used?
yacovm (Wed, 22 Nov 2017 09:12:57 GMT):
its own
jojialex2 (Wed, 22 Nov 2017 11:14:32 GMT):
Has joined the channel.
jojialex2 (Wed, 22 Nov 2017 11:14:35 GMT):
Hi, Please help on the following error :
peer chaincode instantiate using kafka orderer giving following error,
←[34;1mpeer0.org1.example.com |←[0m ←[33m2017-11-22 10:47:31.505 UTC [gossip/comm] sendToEndpoint -> WARN 54d←[0m Failed obtaining connection for 172.18.0.15:7051, PKIid:[71 88 115 27 28 222 244 157 196 20 206 87 238 41 173 71 185 195 175 121 236 34 206 183 63 153 178 63 241 226 254 204] reason: x509: cannot validate certificate for 172.18.0.15 because it doesn't contain any IP SANs
yacovm (Wed, 22 Nov 2017 11:44:50 GMT):
@jojialex2
yacovm (Wed, 22 Nov 2017 11:45:02 GMT):
the problem is that the TLS certificate of your peer, doesn't have IP SANs
yacovm (Wed, 22 Nov 2017 11:46:01 GMT):
you didn't generate it correctly
jojialex2 (Wed, 22 Nov 2017 13:44:09 GMT):
Any reference ?
yacovm (Wed, 22 Nov 2017 14:12:21 GMT):
reference to what?
yacovm (Wed, 22 Nov 2017 14:12:44 GMT):
you can just use hostnames instead of ip addresses
jojialex2 (Thu, 23 Nov 2017 04:14:10 GMT):
Thank you, But I am using hostnames in configtx.yaml file. I am not getting any error for channel, join channel, update anchors, and chaincode install, but this getting while chaincode instantiate. where should I give this hostnames instead of address ?
jojialex2 (Thu, 23 Nov 2017 04:14:10 GMT):
Thank you, But I am using hostnames in configtx.yaml file. I am not getting any error for channel, join channel, update anchors, and chaincode install, but this getting while chaincode instantiate. where should I give this hostnames instead of Ip address ?
jojialex2 (Thu, 23 Nov 2017 04:14:10 GMT):
Thank you, But I am using hostnames in configtx.yaml file. I am not getting any error for channel, join channel, update anchors, and chaincode install, but this getting while chaincode instantiate. where should I give this hostnames instead of ip address ?
yacovm (Thu, 23 Nov 2017 07:43:44 GMT):
You are *not* getting this error at instantiate.
yacovm (Thu, 23 Nov 2017 07:43:54 GMT):
this error is from the gossip logs and orthogonal to instantiate
jojialex2 (Thu, 23 Nov 2017 08:27:28 GMT):
I am not clear, I just start use the hyperledger fabric version 1.0.4
jojialex2 (Thu, 23 Nov 2017 08:27:28 GMT):
I am not clear, I just start useing the hyperledger fabric version 1.0.4
jojialex2 (Thu, 23 Nov 2017 08:27:33 GMT):
this is my file
jojialex2 (Thu, 23 Nov 2017 08:28:12 GMT):
OrdererOrgs:
- Name: ExampleCom
Domain: example.com
Specs:
- Hostname: orderer0
- Hostname: orderer1
- Hostname: orderer2
PeerOrgs:
- Name: Org1ExampleCom
Domain: org1.example.com
Template:
Count: 2
Users:
Count: 1
- Name: Org2ExampleCom
Domain: org2.example.com
Template:
Count: 2
Users:
Count: 1
jojialex2 (Thu, 23 Nov 2017 08:29:03 GMT):
How to resolve this error.
jojialex2 (Thu, 23 Nov 2017 08:29:18 GMT):
x509: cannot validate certificate for 172.18.0.15 because it doesn't contain any IP SANs
yacovm (Thu, 23 Nov 2017 08:29:48 GMT):
what is the full error?
yacovm (Thu, 23 Nov 2017 08:29:49 GMT):
the full line
yacovm (Thu, 23 Nov 2017 08:30:09 GMT):
also what is the CORE_PEER_ADDRESS of the peers in the docker-compose file?
jojialex2 (Thu, 23 Nov 2017 08:30:23 GMT):
I have given
yacovm (Thu, 23 Nov 2017 08:30:37 GMT):
no you have not...
yacovm (Thu, 23 Nov 2017 08:30:49 GMT):
that's the configtx.yaml
jojialex2 (Thu, 23 Nov 2017 08:34:37 GMT):
feature.zip
jojialex2 (Thu, 23 Nov 2017 08:34:53 GMT):
Can you please check ..
yacovm (Thu, 23 Nov 2017 08:36:12 GMT):
what is the while error?
jojialex2 (Thu, 23 Nov 2017 08:36:13 GMT):
feature\configs\onfigtx.yaml
jojialex2 (Thu, 23 Nov 2017 08:36:22 GMT):
feature\configs\configtx.yaml
yacovm (Thu, 23 Nov 2017 08:36:25 GMT):
right right
yacovm (Thu, 23 Nov 2017 08:36:28 GMT):
what is the whole error?
yacovm (Thu, 23 Nov 2017 08:36:33 GMT):
the whole line
jojialex2 (Thu, 23 Nov 2017 08:37:24 GMT):
←[36mpeer1.org1.example.com |←[0m ←[36m2017-11-23 06:05:16.400 UTC [gossip/channel] handleStateInfSnapshot -> DEBU 4a9←[0m Channel businesschannel : Couldn't find or
g identity of peer ∩┐╜∩┐╜4∩┐╜L►Lz∩┐╜∩┐╜t∩┐╜∩┐╜r∩┐╜#e G∩┐╜c∩┐╜⌂∩┐╜⌂`∩┐╜z message sent from ∩┐╜b∩┐╜s∩┐╜∩┐╜∩┐╜∩┐╜Yq&En∩┐╜∩┐╜r&∩┐╜∩┐╜{∩┐╜∩┐╜∩┐╜∩┐╜∩┐╜◄∩┐╜*∩┐╜∩┐╜F∩┐╜
←[34;1mpeer0.org2.example.com |←[0m ←[33m2017-11-23 06:05:28.494 UTC [gossip/comm] sendToEndpoint -> WARN 4cf←[0m Failed obtaining connection for 172.18.0.15:7051, P
KIid:[238 112 58 88 121 252 77 159 184 81 143 231 242 33 108 248 60 2 103 96 44 251 194 35 105 24 236 178 192 233 186 83] reason: x509: cannot validate certificate for
172.18.0.15 because it doesn't contain any IP SANs
jojialex2 (Thu, 23 Nov 2017 08:38:06 GMT):
From Cli
jojialex2 (Thu, 23 Nov 2017 08:38:10 GMT):
[006 11-23 06:05:17.14 UTC] [github.com/hyperledger/fabric/msp] main.Execute.ExecuteC.execute.func1.chaincodeDeploy.instantiate.GetSignedProposal.Sign -> DEBU Sign: pla
intext: 0AB9070A6C08031A0B089DCAD9D00510...65436F6D0A04657363630A0476736363
[007 11-23 06:05:17.14 UTC] [github.com/hyperledger/fabric/msp] main.Execute.ExecuteC.execute.func1.chaincodeDeploy.instantiate.GetSignedProposal.Sign -> DEBU Sign: dig
est: EA73D6034071816EA269C7EE280A8BDE09A5C8DC8D92BEA1A0DCBE7669849FFB
Error: Error endorsing chaincode: rpc error: code = Unknown desc = timeout expired while starting chaincode mycc:1.0(networkid:feature,peerid:peer0.org1.example.com,tx:
e5f11039155d8ea10225655f64ddd17c0230793d966705879f3662aee51e47ca)
jojialex2 (Thu, 23 Nov 2017 09:14:09 GMT):
Please find the log file
jojialex2 (Thu, 23 Nov 2017 09:14:40 GMT):
log.zip
yacovm (Thu, 23 Nov 2017 09:18:23 GMT):
`image: dockersrv02.ad.infosys.com/george.alexander/fabric-peer:x86_64-1.1.0-preview`
yacovm (Thu, 23 Nov 2017 09:18:25 GMT):
what is this?
yacovm (Thu, 23 Nov 2017 09:18:49 GMT):
who is george alexander?
jojialex2 (Thu, 23 Nov 2017 09:18:56 GMT):
this is we pushed into our local hub
jojialex2 (Thu, 23 Nov 2017 09:19:20 GMT):
to avoid multiple down load
yacovm (Thu, 23 Nov 2017 09:19:27 GMT):
ok I have an idea - can you please open a JIRA http://jira.hyperledger.org/
yacovm (Thu, 23 Nov 2017 09:19:38 GMT):
attach to the JIRA issue your zip files
yacovm (Thu, 23 Nov 2017 09:19:43 GMT):
the feature.zip
yacovm (Thu, 23 Nov 2017 09:19:49 GMT):
and write here the JIRA issue
yacovm (Thu, 23 Nov 2017 09:19:55 GMT):
I'll take a look later today when I have time...
jojialex2 (Thu, 23 Nov 2017 09:20:14 GMT):
ok shure ..
jojialex2 (Thu, 23 Nov 2017 09:20:35 GMT):
sorry
jojialex2 (Thu, 23 Nov 2017 09:20:38 GMT):
ok sure
jackeyliliang (Fri, 24 Nov 2017 02:59:06 GMT):
Has joined the channel.
MohitYadav2317 (Fri, 24 Nov 2017 07:10:31 GMT):
Has joined the channel.
jojialex2 (Fri, 24 Nov 2017 11:47:40 GMT):
Hi I am using CLI for chaincode instantiate , once the console get stuck after this message
←[31;1mpeer0.org1.example.com |←[0m ←[36m2017-11-24 11:40:29.631 UTC [dockercontroller] createContainer -> DEBU 3d6←[0m Created container: feature-peer0.org1.example
.com-mycc-1.0-69b833ea5b04412aed03b0935d4ea03b1b0d2b1788b7b308262c5fe39c29eee3
In CLI getting the time out after some time, .
[006 11-24 11:40:29.43 UTC] [github.com/hyperledger/fabric/msp] main.Execute.ExecuteC.execute.func1.chaincodeDeploy.instantiate.GetSignedProposal.Sign -> DEBU Sign: dig
est: 9F86E80C3F14B0F14959DC3781573F95542DE8C2F9E135871804A31E2FD9BFD8
Error: Error endorsing chaincode: rpc error: code = Unknown desc = Timeout expired while starting chaincode mycc:1.0(networkid:feature,peerid:peer0.org1.example.com,tx:
66a909cd6c86b389b73e2760bc86417b9b3047cc85b48f69e86803326a092b1d)
Is any one get this kind of error
Please help
MohitYadav2317 (Fri, 24 Nov 2017 14:40:43 GMT):
hi all, i wanted to know in what use case will anchor peers interact? usually a client calls peers with the proposal and so what exactly is the reason for anchor peer creation?
yacovm (Fri, 24 Nov 2017 16:31:58 GMT):
@MohitYadav2317 anchor peers are to establish cross org membership
egeek (Wed, 29 Nov 2017 11:16:04 GMT):
Has joined the channel.
bh4rtp (Fri, 01 Dec 2017 07:04:35 GMT):
@yacovm what are cross org membership? in physical world, there is no cross org membership.
bh4rtp (Fri, 01 Dec 2017 07:04:35 GMT):
@yacovm what is cross org membership? in physical world, there is no cross org membership.
yacovm (Fri, 01 Dec 2017 07:38:19 GMT):
Why?
ArnabChatterjee (Fri, 01 Dec 2017 08:28:32 GMT):
Has joined the channel.
bizhenchao1201 (Sun, 03 Dec 2017 09:26:23 GMT):
Has joined the channel.
elewis787 (Tue, 05 Dec 2017 17:53:42 GMT):
Has joined the channel.
elewis787 (Tue, 05 Dec 2017 17:57:25 GMT):
Hey all ! I am trying to learn more about blockchain gossip protocols ( and p2p in general ) Looking through some of the gossip code for fabric is seems to be somewhat similar to the raft protocol. Can anyone explain the difference or confirm that it is loosely based on raft ?
yacovm (Tue, 05 Dec 2017 18:35:17 GMT):
It is nowhere near raft
elewis787 (Tue, 05 Dec 2017 18:38:30 GMT):
mind explain a little more why ? I have been reading through the gossip/election code. The pseudo code for identify a leader is what got me thinking about raft.
yacovm (Tue, 05 Dec 2017 19:15:13 GMT):
oh that
yacovm (Tue, 05 Dec 2017 19:15:37 GMT):
so, the main difference is... that the LE in gossip, favors availability / progress over consistency
yacovm (Tue, 05 Dec 2017 19:15:48 GMT):
in other words, if you have a partition, some leader will get elected
yacovm (Tue, 05 Dec 2017 19:16:13 GMT):
even if the partition is you as a peer, is isolated from the rest - you'll simply elect yourself as a leader
yacovm (Tue, 05 Dec 2017 19:16:25 GMT):
@elewis787 ^
elewis787 (Tue, 05 Dec 2017 19:18:33 GMT):
I see. So the actual data dissemination is weakly consistent meaning some nodes may be out of sync ?
yacovm (Tue, 05 Dec 2017 19:21:29 GMT):
no no no
yacovm (Tue, 05 Dec 2017 19:21:34 GMT):
this has nothing to do with it
yacovm (Tue, 05 Dec 2017 19:21:46 GMT):
the leader election only determines who will connect to the orderer
yacovm (Tue, 05 Dec 2017 19:21:48 GMT):
that's all
yacovm (Tue, 05 Dec 2017 19:22:06 GMT):
the peers want only 1 peer to connect to the orderer, from all the organization
yacovm (Tue, 05 Dec 2017 19:22:11 GMT):
and the rest to receive the blocks via gossip
yacovm (Tue, 05 Dec 2017 19:23:01 GMT):
of course the ledger, is identical among all peers (strong consistency in that aspect)
yacovm (Tue, 05 Dec 2017 19:23:06 GMT):
otherwise it wouldn't be a blockchain right?
elewis787 (Tue, 05 Dec 2017 19:25:59 GMT):
I see. That makes sense thanks for the explanation !
samwood (Tue, 05 Dec 2017 20:12:13 GMT):
Has joined the channel.
samwood (Tue, 05 Dec 2017 20:21:30 GMT):
@yacovm can you please expand on that? If the blocks are disseminated via gossip which is not strongly consistent, then how is the ledger identical among all the peers?
yacovm (Tue, 05 Dec 2017 20:23:53 GMT):
The leader election isn't strongly consistent
yacovm (Tue, 05 Dec 2017 20:23:58 GMT):
the replication is
yacovm (Tue, 05 Dec 2017 20:24:15 GMT):
each block, gets put into the ledger by order
yacovm (Tue, 05 Dec 2017 20:24:19 GMT):
and is verified
samwood (Tue, 05 Dec 2017 20:27:30 GMT):
does the leader wait until the other peers have received the block before committing it?
yacovm (Tue, 05 Dec 2017 20:29:58 GMT):
no...
samwood (Tue, 05 Dec 2017 21:15:45 GMT):
I'm having trouble seeing how it's strongly consistent. Suppose there are 2 peers, p1 and p2. Suppose a client queries a CC for key "k", which has value "v1" at time t1. Specifically, suppose query(p1, k)@t1 = v1, and query(p2, k)@t1 = v1. Now suppose "k" has been updated to "v2", and p1 is the leader. Could there not be a race where query(p1, k)@t2 = v2 and query(p2, k)@t2 = v1?
samwood (Tue, 05 Dec 2017 21:15:45 GMT):
I'm having trouble seeing how it's strongly consistent. Suppose there are 2 peers, p1 and p2. Suppose a client queries a CC for key "k", which has value "v1" at time t1. Specifically, suppose query(p1, k)@t1 = v1, and query(p2, k)@t1 = v1. Now suppose "k" has been updated to "v2", and p1 is the leader. Could there not be a race where query(p1, k)@t2 = v2 and query(p2, k)@t3 = v1?
C0rWin (Tue, 05 Dec 2017 22:09:53 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=K3Nmq7eDiQad8cTHu) @samwood Leader means being responsible to pull blocks with ordered transactions from the ordering service and take care of blocks dissemination within the org, I'm not sure how this related to the consistency and scenario you've described? also as @yacovm mentioned, election isn't strongly consistent, e.g. there could be a case where an org will have more than one leader, but in that case the only implication will be that two peers pulling blocks from the ordering instead of one.
C0rWin (Tue, 05 Dec 2017 22:11:42 GMT):
and to your point, yes it's possible that if ledger height on two different peer not the same, you might query and see different results
C0rWin (Tue, 05 Dec 2017 22:12:07 GMT):
although, this has no relation to the gossip leader election
samwood (Tue, 05 Dec 2017 22:12:31 GMT):
@C0rWin sure, but this is a result of the block replication which seems to be eventually consistent, correct?
C0rWin (Tue, 05 Dec 2017 22:12:57 GMT):
what is the result of block replication?
C0rWin (Tue, 05 Dec 2017 22:13:37 GMT):
generally speaking, yes, state update is eventually consistent
samwood (Tue, 05 Dec 2017 22:13:38 GMT):
I'm having trouble reconciling your comment "yes it's possible that if ledger height on two different peer not the same, you might query and see different results " , with yacovm's earlier comment "of course the ledger, is identical among all peers (strong consistency in that aspect) "
C0rWin (Tue, 05 Dec 2017 22:14:23 GMT):
well, there is a latency in the network
C0rWin (Tue, 05 Dec 2017 22:14:49 GMT):
so it might be the case where peer p1 at ledger height 100, while peer p2 at 99
C0rWin (Tue, 05 Dec 2017 22:15:16 GMT):
wrt, state up until block 99, both ledger are equal
C0rWin (Tue, 05 Dec 2017 22:15:16 GMT):
wrt state, up until block 99, both ledger are equal
C0rWin (Tue, 05 Dec 2017 22:15:16 GMT):
wrt state, up until block 99, both ledgers are equal
C0rWin (Tue, 05 Dec 2017 22:17:17 GMT):
also blocks applied and delivered to the ledger according to the sequence number
C0rWin (Tue, 05 Dec 2017 22:17:59 GMT):
so say peer p2, from my example which at block num 99 will get block 110, it will still wait until it will get block number 100
samwood (Tue, 05 Dec 2017 22:21:38 GMT):
OK, thanks. Do leaders commit their blocks as soon as they receive them from the orderer? Just to clarify, they don't communicate with other leaders or peers before committing blocks correct? (e.g, a 2 phase commit)
C0rWin (Tue, 05 Dec 2017 22:23:24 GMT):
there is no need for 2 phase commit here
C0rWin (Tue, 05 Dec 2017 22:23:48 GMT):
leader commits the block as soon as it gets it
C0rWin (Tue, 05 Dec 2017 22:24:42 GMT):
what will be the reason to communicate with other leaders?
C0rWin (Tue, 05 Dec 2017 22:24:57 GMT):
block signed with OSN
C0rWin (Tue, 05 Dec 2017 22:25:09 GMT):
could be easily verified
samwood (Tue, 05 Dec 2017 22:25:16 GMT):
sorry, what's OSN?
C0rWin (Tue, 05 Dec 2017 22:25:23 GMT):
Ordering Service Node
yacovm (Tue, 05 Dec 2017 22:29:05 GMT):
To be honest I don't think that strong vs weak consistency are right terminologies to describe peers.
yacovm (Tue, 05 Dec 2017 22:30:21 GMT):
Usually when you talk about strong vs weak you think about replicated state machines vs. systems that may return a false result but eventually converge
yacovm (Tue, 05 Dec 2017 22:30:47 GMT):
however - in fabric, each time you query a peer (assuming it is honest) - it will return to you a result that was correct to some point in time
yacovm (Tue, 05 Dec 2017 22:30:53 GMT):
it may not be the latest result
yacovm (Tue, 05 Dec 2017 22:31:04 GMT):
but it is always "correct"
yacovm (Tue, 05 Dec 2017 22:31:42 GMT):
it just may be in the past, and the peer isn't updated yet
yacovm (Tue, 05 Dec 2017 22:31:59 GMT):
Even in replicated state machines, a replica or two might be lagging behind
yacovm (Tue, 05 Dec 2017 22:32:40 GMT):
the difference is that in fabric, most peers might be lagging behind
yacovm (Tue, 05 Dec 2017 22:32:51 GMT):
its because the consensus isn't done in the peers
samwood (Tue, 05 Dec 2017 22:36:36 GMT):
thanks for the explanations @C0rWin @yacovm
samwood (Tue, 05 Dec 2017 22:39:51 GMT):
I agree that the strong/weak consistency terminology is confusing. |When I think of strong consistency I think that all peers will agree on a value before any of them return it to a client. E..g, all peers appear to be at the same height from the perspective of the client
yacovm (Tue, 05 Dec 2017 22:41:34 GMT):
So in that case it is not
samwood (Tue, 05 Dec 2017 22:57:43 GMT):
i gather the orderer maintains a current block height and rejects transactions that read data from a block height that is less than the current block height (per the "read-write set semantics" section in the docs) ?
yacovm (Tue, 05 Dec 2017 22:58:35 GMT):
No, orderers blidly order transactions
yacovm (Tue, 05 Dec 2017 22:58:39 GMT):
*blindly
yacovm (Tue, 05 Dec 2017 22:59:00 GMT):
You throw trnsactions into them and they cut identical blocks
yacovm (Tue, 05 Dec 2017 22:59:10 GMT):
(identical across all orderers)
yacovm (Tue, 05 Dec 2017 22:59:20 GMT):
They are strongly consistant :wink:
yacovm (Tue, 05 Dec 2017 22:59:58 GMT):
Though, it is also not a good way of describing
yacovm (Tue, 05 Dec 2017 23:00:09 GMT):
Bevause the strong consistency comes from kafka
samwood (Tue, 05 Dec 2017 23:00:26 GMT):
hmm, just thinking out loud here, but if the orderer isn't enforcing the read-write set semantics and the peers aren't strongly consistent (per discussion above)) then are there not situations then that could put the network in an inconsistent state?
yacovm (Tue, 05 Dec 2017 23:00:36 GMT):
An orderer node might be behind too
yacovm (Tue, 05 Dec 2017 23:00:51 GMT):
If he cant connect to kafka
yacovm (Tue, 05 Dec 2017 23:01:34 GMT):
So, the read-write logical semantics are enforced at commit
samwood (Tue, 05 Dec 2017 23:07:09 GMT):
i guess i'm confused what commit means -- this is after the tx has been endorsed, and included in a block by the orderer?
samwood (Tue, 05 Dec 2017 23:09:13 GMT):
is it possible for 2 peers to be at the same height and have different world state ?
yacovm (Tue, 05 Dec 2017 23:11:43 GMT):
It is not possible
yacovm (Tue, 05 Dec 2017 23:12:08 GMT):
Yes, commit is when the transaction comes into the ledger
C0rWin (Tue, 05 Dec 2017 23:15:08 GMT):
c
yacovm (Tue, 05 Dec 2017 23:17:16 GMT):
@samwood https://hyperledger-fabric.readthedocs.io/en/latest/arch-deep-dive.html
yacovm (Tue, 05 Dec 2017 23:17:50 GMT):
read up to 4 non-including
crozzy (Tue, 05 Dec 2017 23:26:48 GMT):
Has joined the channel.
manish-sethi (Tue, 05 Dec 2017 23:49:40 GMT):
Has left the channel.
ngeorge (Thu, 07 Dec 2017 11:40:03 GMT):
Has joined the channel.
ngeorge (Thu, 07 Dec 2017 11:41:00 GMT):
Hi all, in the fabric docs it was given "Anchor peer is a peer node on a channel that all other peers can discover and communicate with." My understanding is that the communication between peers of different orgs is through these anchor peers. To test that, I brought down the anchor peer of OrgA and triggered an invoke transaction on a peer of another Org. Later queried the non anchor peer of OrgA and I could still get the latest result for query. So, what exactly means by " all other peers can discover and communicate with"?
mbharadwaj (Thu, 07 Dec 2017 11:44:33 GMT):
Has joined the channel.
yacovm (Thu, 07 Dec 2017 11:55:46 GMT):
so the peers learn about peers of other organization
yacovm (Thu, 07 Dec 2017 11:55:54 GMT):
by contacting the anchor peer(s)
yacovm (Thu, 07 Dec 2017 11:56:01 GMT):
and then they exchange information about peers
JayJong (Thu, 07 Dec 2017 12:23:02 GMT):
Has joined the channel.
PetrVlasekCA (Fri, 08 Dec 2017 10:19:00 GMT):
Has joined the channel.
Lakshmipadmaja (Fri, 08 Dec 2017 14:43:45 GMT):
Has joined the channel.
alvaradojl (Sat, 09 Dec 2017 16:23:20 GMT):
Has joined the channel.
hayato (Sun, 10 Dec 2017 12:01:52 GMT):
Has joined the channel.
guolidong (Tue, 12 Dec 2017 05:58:11 GMT):
Has joined the channel.
Khaled.MH (Wed, 13 Dec 2017 14:56:39 GMT):
Has joined the channel.
david_dornseifer (Thu, 14 Dec 2017 00:38:49 GMT):
Has joined the channel.
asaningmaxchain (Thu, 14 Dec 2017 02:04:02 GMT):
@yacovm i use the orderer,peer binary to start the node, and i create the channel and then join the channel ,but when i join, i find a problem.can you give me some clue?
asaningmaxchain (Thu, 14 Dec 2017 02:04:55 GMT):
https://pastebin.com/aRhZw25E this is the peer info log
yacovm (Thu, 14 Dec 2017 08:03:37 GMT):
well it seems you can't connect to the orderer.example
yacovm (Thu, 14 Dec 2017 08:03:50 GMT):
can you try to get into the container and lookup the hostname?
yacovm (Thu, 14 Dec 2017 08:03:55 GMT):
maybe it doesn't know it or something
ArnabChatterjee (Mon, 18 Dec 2017 12:12:00 GMT):
Hi @yacovm . Can you let me know why are we not joining anchor peers in the channel and the balance-transfer example? I want to know the benefit and disadvantage if any of defining anchor peers. Also, please let me know what is the effect of not having anchor peers on election or behavior of leading peer. Thank you.
C0rWin (Mon, 18 Dec 2017 12:21:15 GMT):
@ArnabChatterjee you need anchor peers to be able to connect two different organization, think of it like a cross org bootstrap set, it's orthogonal to the concept of leader election and presence or absence of anchors has no effect on leader election
C0rWin (Mon, 18 Dec 2017 12:22:28 GMT):
leader election purpose is to choose the peer responsible to get connected to the ordering service and pull the blocks via gossip to rest of the peers
C0rWin (Mon, 18 Dec 2017 12:23:09 GMT):
peers monitoring availability of the leader peer and in case it's down or not reachable for some reason a new leader has been elected
C0rWin (Mon, 18 Dec 2017 12:23:44 GMT):
while wrt anchor peers, you need them so two organization will be able to advertise and share membership of peers within the channel
ArnabChatterjee (Mon, 18 Dec 2017 12:31:14 GMT):
Thanks @C0rWin . How does setting anchor peer enable other orgs to discover peers in my org? Does it somehow expose my endorsing peers to allow for endorsement or something like that?
C0rWin (Mon, 18 Dec 2017 12:31:41 GMT):
no
C0rWin (Mon, 18 Dec 2017 12:31:56 GMT):
setting anchor peers and config update transaction
C0rWin (Mon, 18 Dec 2017 12:31:56 GMT):
setting anchor peers is a config update transaction
C0rWin (Mon, 18 Dec 2017 12:32:24 GMT):
once it get applied within orgs, anchors peers get learned and used to bridge the membership across peers
C0rWin (Mon, 18 Dec 2017 12:33:03 GMT):
peers in fact shares local membership views via anchors
zhishui (Tue, 19 Dec 2017 08:10:36 GMT):
Has joined the channel.
EricYang (Wed, 03 Jan 2018 06:48:00 GMT):
Has joined the channel.
gurel (Wed, 03 Jan 2018 17:45:46 GMT):
Has joined the channel.
allonblocks21 (Thu, 04 Jan 2018 11:06:16 GMT):
Has joined the channel.
sykesm (Fri, 05 Jan 2018 18:58:57 GMT):
Has joined the channel.
sykesm (Fri, 05 Jan 2018 19:22:41 GMT):
Hi. I'm new to the code base and looking at ways to speed up some of the unit test coverage. I've noticed there are a lot of sleeps in the gossip/comm package. One of the patterns in the tests it to setup two comm instances, start go routines to handle messages, and then send messages across the comm instances. Between the two calls to send, there's a sleep.
When the sleep is removed, the tests fail. It looks like this happens because of a race between client connections and server connections. Basically, client connections from a->b created by `Send` are closed (and the associated output buffer with the pending message is drained) when the connection from b->a is established and the `onConnected` callback is driven.
The sleep is clearly avoiding this issue but is losing the message the correct behavior? Should everything in the send queue be dropped on the floor when the message target initiates a connection concurrently?
yacovm (Fri, 05 Jan 2018 19:27:09 GMT):
uh, I don't think it has to do with queues, actually.
yacovm (Fri, 05 Jan 2018 19:27:19 GMT):
i'ts basically because gossip keeps a single connection to a peer
yacovm (Fri, 05 Jan 2018 19:27:42 GMT):
so if you connect to someone and someone connects to you at the same time...
yacovm (Fri, 05 Jan 2018 19:28:09 GMT):
you might "succeed" but then the connection is closed on the other side
yacovm (Fri, 05 Jan 2018 19:28:16 GMT):
because he did the same thing
sykesm (Fri, 05 Jan 2018 19:28:42 GMT):
So, help me out. TestBasic in gossip/comm/comm_test.go
sykesm (Fri, 05 Jan 2018 19:29:02 GMT):
I'm guessing it's the "most basic" flow that exists.
sykesm (Fri, 05 Jan 2018 19:29:23 GMT):
the `onConnection` callback happening when b connects back to a is closing the connection from a to b
sykesm (Fri, 05 Jan 2018 19:29:40 GMT):
I get *why* that's happening.
sykesm (Fri, 05 Jan 2018 19:29:40 GMT):
I get *why* that's happening. (single connection between peers is desired)
sykesm (Fri, 05 Jan 2018 19:30:42 GMT):
going back to the goal (remove the sleeps), what is expected?
yacovm (Fri, 05 Jan 2018 19:30:46 GMT):
if you want to make tests faster I don't think the comm package is the right place, actually
yacovm (Fri, 05 Jan 2018 19:30:52 GMT):
there are much worse places :(
yacovm (Fri, 05 Jan 2018 19:30:56 GMT):
like gossip/service
sykesm (Fri, 05 Jan 2018 19:31:11 GMT):
I'm starting somewhere - new to code so i picked some places.
sykesm (Fri, 05 Jan 2018 19:31:13 GMT):
This is one.
yacovm (Fri, 05 Jan 2018 19:31:49 GMT):
> going back to the goal (remove the sleeps), what is expected?
what do you mean what is expected?
sykesm (Fri, 05 Jan 2018 19:32:31 GMT):
The test is structured where it's sending a message, sleeping, and sending a message. Should it be structured (instead) to send a message and simply expect the target to receive it?
yacovm (Fri, 05 Jan 2018 19:33:05 GMT):
oh,
sykesm (Fri, 05 Jan 2018 19:33:08 GMT):
Is dropping the contents of the send queue expected? (possibly - since it's no blocking)
yacovm (Fri, 05 Jan 2018 19:33:24 GMT):
no, the point was to to have both instances send each other
yacovm (Fri, 05 Jan 2018 19:33:51 GMT):
the sleep between is because the code doesn't support that concurrent event :/
sykesm (Fri, 05 Jan 2018 19:34:06 GMT):
but the test is artificially working around the race for a positive outcome - and there are other ways that take less real time
yacovm (Fri, 05 Jan 2018 19:34:06 GMT):
when 2 peers create connections, at the very same instance
yacovm (Fri, 05 Jan 2018 19:34:21 GMT):
yeah there are
yacovm (Fri, 05 Jan 2018 19:34:23 GMT):
we can make the test
yacovm (Fri, 05 Jan 2018 19:34:38 GMT):
check if the 2nd peer got the message from the 1st
yacovm (Fri, 05 Jan 2018 19:34:43 GMT):
and then continue
sykesm (Fri, 05 Jan 2018 19:34:44 GMT):
that's what I did.
sykesm (Fri, 05 Jan 2018 19:34:48 GMT):
went down to ~4ms
yacovm (Fri, 05 Jan 2018 19:34:49 GMT):
ok
yacovm (Fri, 05 Jan 2018 19:35:06 GMT):
cool we've gained a second :)
sykesm (Fri, 05 Jan 2018 19:35:29 GMT):
but I wanted to ask explicitly about the fact that the onConnection happening on the server - it closes the client connection and drops all pending messages
sykesm (Fri, 05 Jan 2018 19:35:46 GMT):
that really seems odd - we didn't "drain" it by sending - we dropped them
sykesm (Fri, 05 Jan 2018 19:36:01 GMT):
local tests are down to 6m for me (so it's not just this)
yacovm (Fri, 05 Jan 2018 19:36:29 GMT):
yes we did
yacovm (Fri, 05 Jan 2018 19:36:58 GMT):
well this is because implementing detection that a connections happens from the other side while you are connecting to that side is much more complex
yacovm (Fri, 05 Jan 2018 19:37:10 GMT):
to just have a naive implementation that replaces the connection always
yacovm (Fri, 05 Jan 2018 19:39:57 GMT):
but don't you think it's more cost-effective to go after "slow" tests (i.e the service package, they are not parallelized)?
yacovm (Fri, 05 Jan 2018 19:40:20 GMT):
than to re-write a test that takes 1 second :neutral_face:
yacovm (Fri, 05 Jan 2018 19:40:47 GMT):
what do you mean down to 6m?
sykesm (Fri, 05 Jan 2018 19:41:27 GMT):
I think you've answered my question so I'll swizzle the tests.
sykesm (Fri, 05 Jan 2018 19:42:02 GMT):
What do I mean? I mean with changes I've made locally in the unit test scripts, simple parallel execution across packages has brought the unit test execution to under 6m.
yacovm (Fri, 05 Jan 2018 19:42:19 GMT):
the entire UT?
sykesm (Fri, 05 Jan 2018 19:42:28 GMT):
Now I'm going package by package for the long poles
yacovm (Fri, 05 Jan 2018 19:42:30 GMT):
the entire UT suite?
sykesm (Fri, 05 Jan 2018 19:42:31 GMT):
gossip is along pole
sykesm (Fri, 05 Jan 2018 19:42:38 GMT):
yes. why does that surprise?
yacovm (Fri, 05 Jan 2018 19:42:54 GMT):
because only the compilation time of each of them is very high
yacovm (Fri, 05 Jan 2018 19:43:21 GMT):
ah you made them run concurrently
sykesm (Fri, 05 Jan 2018 19:43:23 GMT):
sure - because right now they get built package by package and the intermediate isn't saved
yacovm (Fri, 05 Jan 2018 19:43:37 GMT):
ah I tried this in the past... but the CI server couldn't handle the stress :(
sykesm (Fri, 05 Jan 2018 19:43:42 GMT):
if you actually let the go tooling do what it's supposed to do, life would be good
yacovm (Fri, 05 Jan 2018 19:43:46 GMT):
it worked on my machine but failed in CI
sykesm (Fri, 05 Jan 2018 19:44:17 GMT):
i need to get with ramesh at some point - code coverage is the other nit
sykesm (Fri, 05 Jan 2018 19:44:27 GMT):
fixed in go 1.10 - but that's a month or two out
yacovm (Fri, 05 Jan 2018 19:45:59 GMT):
ok
sykesm (Fri, 05 Jan 2018 19:48:36 GMT):
fwiw, bccsp/{factory,pkcs11} (already did sw), core/deliverservice{,blocksprovider},
yacovm (Fri, 05 Jan 2018 19:49:16 GMT):
yeah the problem with BCCSP is that it's CPU-intensive but it's not parallel IIRC
yacovm (Fri, 05 Jan 2018 19:49:28 GMT):
so if you parallelize it it should run faster
sykesm (Fri, 05 Jan 2018 19:49:34 GMT):
I addressed sw to make it parallel
yacovm (Fri, 05 Jan 2018 19:49:41 GMT):
makes sense :)
sykesm (Fri, 05 Jan 2018 19:50:21 GMT):
that was last month. port reuse in gossip is what brought me down this path in parallel - so I'll get back to what I was doing. thanks for the answer.
yacovm (Fri, 05 Jan 2018 19:51:48 GMT):
port resuse?
yacovm (Fri, 05 Jan 2018 19:51:54 GMT):
*reuse
yacovm (Fri, 05 Jan 2018 19:52:02 GMT):
we use different ports for everything
yacovm (Fri, 05 Jan 2018 19:52:02 GMT):
we use different ports for every test
yacovm (Fri, 05 Jan 2018 19:52:10 GMT):
except... in gossip/service I think
yacovm (Fri, 05 Jan 2018 19:52:30 GMT):
@sykesm
sykesm (Fri, 05 Jan 2018 19:56:28 GMT):
perhaps. but
```
--- FAIL: TestAccept (0.01s)
panic: listen tcp :7611: bind: address already in use [recovered]
panic: listen tcp :7611: bind: address already in use
...
--- FAIL: TestSendByCriteria (0.02s)
panic: listen tcp :20000: bind: address already in use [recovered]
panic: listen tcp :20000: bind: address already in use
...
```
yacovm (Fri, 05 Jan 2018 19:56:51 GMT):
odd
yacovm (Fri, 05 Jan 2018 19:57:06 GMT):
how come this never happens in CI?
yacovm (Fri, 05 Jan 2018 19:57:12 GMT):
or on my machine
sykesm (Fri, 05 Jan 2018 19:57:12 GMT):
nothing runs in parallel in ci
yacovm (Fri, 05 Jan 2018 19:57:18 GMT):
ohhhhh
yacovm (Fri, 05 Jan 2018 19:57:28 GMT):
you run tests from different packages in parallel
yacovm (Fri, 05 Jan 2018 19:57:54 GMT):
anyway, it would be great if CI time would be reduced
yacovm (Fri, 05 Jan 2018 19:58:05 GMT):
so appreciate your efforts in trying
sykesm (Fri, 05 Jan 2018 19:58:19 GMT):
I simply use go the way it was intended to be used `go test ./...`
yacovm (Fri, 05 Jan 2018 19:58:32 GMT):
this runs everything in parallel?
sykesm (Fri, 05 Jan 2018 19:58:34 GMT):
yes, I'm finding all kinds of other issues
sykesm (Fri, 05 Jan 2018 19:58:49 GMT):
it runs packages in parallel, not tests in the package
sykesm (Fri, 05 Jan 2018 19:58:58 GMT):
but some of the tests are marked as parallel as well (not done by me)
yacovm (Fri, 05 Jan 2018 19:59:10 GMT):
back when I tried, I simply made a script that batches them into `K` groups for a configurable `K` but I encountered timeouts in tests
sykesm (Fri, 05 Jan 2018 19:59:11 GMT):
but there are other issues - code that assumes link flags are set
yacovm (Fri, 05 Jan 2018 19:59:12 GMT):
so I dropped it
sykesm (Fri, 05 Jan 2018 20:00:04 GMT):
go tests runs n commands/test binaries in parallel. those commands can be compilation or test binaries or linkers or whatever. by default 'n' is MAXGOPROCS.
yacovm (Fri, 05 Jan 2018 20:00:20 GMT):
and does it run them in the same process
yacovm (Fri, 05 Jan 2018 20:00:24 GMT):
or different proceses?
yacovm (Fri, 05 Jan 2018 20:00:24 GMT):
or different processes?
sykesm (Fri, 05 Jan 2018 20:00:35 GMT):
in ci, -p 1 is explcitly set in one build time and in the other they explicitly run package by package
sykesm (Fri, 05 Jan 2018 20:00:35 GMT):
in ci, -p 1 is explcitly set in one build and in the other they explicitly run package by package
sykesm (Fri, 05 Jan 2018 20:00:53 GMT):
different processes
yacovm (Fri, 05 Jan 2018 20:00:56 GMT):
cool
yacovm (Fri, 05 Jan 2018 20:01:13 GMT):
so a hurdle you'll likely experience is also the ledger tests, I think
yacovm (Fri, 05 Jan 2018 20:01:17 GMT):
they write to the file system
sykesm (Fri, 05 Jan 2018 20:01:26 GMT):
I'll find things all over. that doesn't scare me
sykesm (Fri, 05 Jan 2018 20:01:46 GMT):
I just can't work on a project that takes an hour to build. I won't.
DannyWong (Tue, 09 Jan 2018 03:55:57 GMT):
Has left the channel.
tamycova (Tue, 09 Jan 2018 11:40:06 GMT):
Has joined the channel.
ArnabChatterjee (Fri, 12 Jan 2018 00:50:41 GMT):
Hi. I wanted to know how faulty peer nodes are brought back in sync using gossip. Any ideas?
ArnabChatterjee (Fri, 12 Jan 2018 00:54:59 GMT):
Specifically, I wanted to know more about the mentioned "anti-entropy" mechanism. Thanks. @yacovm @C0rWin
SjirNijssen (Mon, 15 Jan 2018 08:46:31 GMT):
Has joined the channel.
olrraju (Tue, 16 Jan 2018 01:52:29 GMT):
Has joined the channel.
qylixin (Wed, 17 Jan 2018 06:08:31 GMT):
Has joined the channel.
javrevasandeep (Wed, 17 Jan 2018 08:22:24 GMT):
Has joined the channel.
SimonOberzan (Wed, 17 Jan 2018 12:54:50 GMT):
Has left the channel.
mogamboizer (Wed, 17 Jan 2018 23:47:33 GMT):
Has joined the channel.
mogamboizer (Wed, 17 Jan 2018 23:51:23 GMT):
How can I monitor gossip traffic between dockerized peers running on the same host, any gui-tool? :mag:
mogamboizer (Wed, 17 Jan 2018 23:51:23 GMT):
How can I *monitor gossip traffic* between dockerized peers running on the same host, any gui-tool? :mag:
C0rWin (Thu, 18 Jan 2018 07:38:54 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=S96j423RZ9Ahtx8po) @mogamboizer wireshark?
yacovm (Thu, 18 Jan 2018 08:34:01 GMT):
or alternatively turn on gossip logs to debug
yacovm (Thu, 18 Jan 2018 08:34:20 GMT):
`CORE_LOGGING_GOSSIP=DEBUG`
Ammu (Thu, 18 Jan 2018 13:36:23 GMT):
Has joined the channel.
mogamboizer (Thu, 18 Jan 2018 16:48:48 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=xmRhnCq5q6P8M9e9o) @yacovm Thanks that is useful.
mogamboizer (Thu, 18 Jan 2018 16:50:52 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=r4WJTMXT2XsBG7bqZ) @C0rWin any tips on how to best setup wireshark filtering on the docker host? I used the "any" interface and see a lot of encrypted traffic.
yacovm (Thu, 18 Jan 2018 16:51:33 GMT):
may I ask why you want to see the raw traffic?
yacovm (Thu, 18 Jan 2018 16:51:39 GMT):
it's protobuf... it's binary anyway
yacovm (Thu, 18 Jan 2018 16:51:47 GMT):
it's not human-readable like HTTP
yacovm (Thu, 18 Jan 2018 16:51:55 GMT):
you need tools and the proto schema to see that anyway
yacovm (Thu, 18 Jan 2018 16:52:10 GMT):
which is, kinda worthless since the gossip logs, well - print in debug the messages anyway
mogamboizer (Thu, 18 Jan 2018 16:56:18 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=BHXwT4KM4T8q9TevW) @yacovm just curious to see the exchange, didn't setup CORE_LOGGING_GOSSIP yet I assume that will show the messaging details.
yacovm (Thu, 18 Jan 2018 16:56:27 GMT):
yes
jyellick (Thu, 18 Jan 2018 19:09:45 GMT):
@yacovm Do peers ever gossip blocks across organizations? Or is that reserved strictly for things like private data?
yacovm (Thu, 18 Jan 2018 19:10:28 GMT):
not via classic gossip but via point-to-point state transfer, and only if they were already committed
yacovm (Thu, 18 Jan 2018 19:10:48 GMT):
private data is transferred in 2 ways:
yacovm (Thu, 18 Jan 2018 19:11:29 GMT):
1) When state transfer (block transfer which is cross-org too) occurs, the peer piggybacks related private data on these messages
yacovm (Thu, 18 Jan 2018 19:12:05 GMT):
2) When the gossip coordinator (`gossip/privdata/coordinator.go`) detects missing private data that it doesn't have on disk, it fetches from other peers
yacovm (Thu, 18 Jan 2018 19:12:14 GMT):
for the latter its also cross org
yacovm (Thu, 18 Jan 2018 19:12:18 GMT):
but it's only private data
yacovm (Thu, 18 Jan 2018 19:12:26 GMT):
and we ask the MSP to check if it's eligible
yacovm (Thu, 18 Jan 2018 19:12:34 GMT):
I guess you're asking for config changes?
yacovm (Thu, 18 Jan 2018 19:12:46 GMT):
Note that gossip doesn't use the ACL management
yacovm (Thu, 18 Jan 2018 19:12:51 GMT):
only channel config
yacovm (Thu, 18 Jan 2018 19:14:25 GMT):
oh I also forgot
yacovm (Thu, 18 Jan 2018 19:14:31 GMT):
that peers also send private data upon endorsement
yacovm (Thu, 18 Jan 2018 19:14:36 GMT):
according to latest config
jyellick (Thu, 18 Jan 2018 19:14:42 GMT):
It was actually a more generic question I was dm-ed, since I did not know the answer, I thought I would ask here.
> not via classic gossip but via point-to-point state transfer, and only if they were already committed
Just to confirm, for 'classic gossip', blocks are gossiped speculatively (a peer may re-broadcast a block to another peer, prior to the commitment time validation checks).
For "point to point state transfer", this would be a peer responding to an explicit request for a block? Will this happen automatically if an org is partitioned from the ordering service (but not to another peer org which has a connection to the orderer)
yacovm (Thu, 18 Jan 2018 19:15:02 GMT):
yes and yes
jyellick (Thu, 18 Jan 2018 19:15:22 GMT):
Great, thanks @yacovm !
yacovm (Thu, 18 Jan 2018 19:15:38 GMT):
> this would be a peer responding to an explicit request for a block?
The peer detects a fellow peer that has a higher ledger height than his
yacovm (Thu, 18 Jan 2018 19:15:48 GMT):
and thus asks for a block from the highest one
yacovm (Thu, 18 Jan 2018 19:15:48 GMT):
and then asks for a block from the highest one
jyellick (Thu, 18 Jan 2018 19:17:30 GMT):
One final question -- does the point-to-point state transfer involve the gossip 'leader'? Or is 'leader' only related to the ordering service?
yacovm (Thu, 18 Jan 2018 19:17:45 GMT):
the latter :)
jyellick (Thu, 18 Jan 2018 19:18:00 GMT):
Perfect, thanks again!
yacovm (Thu, 18 Jan 2018 19:18:03 GMT):
sure, anytime
B2BProgrammer (Fri, 19 Jan 2018 08:10:14 GMT):
Has joined the channel.
ArnabChatterjee (Fri, 19 Jan 2018 10:13:59 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=GgSCsqQMb6kfZWPCd) Any ideas @yacovm @C0rWin
ArnabChatterjee (Fri, 19 Jan 2018 10:13:59 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=GgSCsqQMb6kfZWPCd) Any ideas @yacovm @C0rWin .Specifically, I wanted to know more about the mentioned "anti-entropy" mechanism.
C0rWin (Fri, 19 Jan 2018 14:14:06 GMT):
Do you have specific question?
C0rWin (Fri, 19 Jan 2018 14:14:32 GMT):
Anti entropy is a common technique in gossip leterature
Brucepark (Sat, 20 Jan 2018 06:14:48 GMT):
Has joined the channel.
Ammu (Sat, 20 Jan 2018 11:54:20 GMT):
can anyone help for this error
Ammu (Sat, 20 Jan 2018 11:54:20 GMT):
chaincode.png
C0rWin (Sat, 20 Jan 2018 12:15:42 GMT):
Please ask in #fabric @Ammu
anishman (Tue, 23 Jan 2018 00:58:37 GMT):
Has joined the channel.
anishman (Tue, 23 Jan 2018 01:33:04 GMT):
Hello everyone, I have a question regarding the sync of peer node(s) once they are stopped and restarted. Is there some specific standard logs in the peer container that confirms that the peer has been brought back to sync with the remaining peers? I'm looking at the peer container logs but to no avail. Thanks beforehand.
anishman (Tue, 23 Jan 2018 01:33:04 GMT):
Hello everyone, I have a question regarding the sync of peer node(s) once they are stopped and restarted. Is there some specific standard logs in the peer container that confirms that the peer has been brought back to sync with the remaining peers? I'm looking at the peer container logs but to no avail. Thanks beforehand. @yacovm
ArnabChatterjee (Tue, 23 Jan 2018 04:13:52 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=8HBjXzvRfYeX3v2MX) @C0rWin - Thanks. I want to know if there is any specific mechanism that is carried out. For example, if I am having 4 peer nodes and 3 of them are faultuy, is there any kind of implicit consensus running at the gossip level that will come into action and bring all 4 of them in sync. I am interested in knowing if something like BFT is implemented?
ArnabChatterjee (Tue, 23 Jan 2018 04:13:52 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=8HBjXzvRfYeX3v2MX) @C0rWin - Thanks. I want to know if there is any specific mechanism that is carried out. For example, if I am having 4 peer nodes and 3 of them are faultuy, is there any kind of implicit consensus running at the gossip level that will come into action and bring all 4 of them in sync. I am interested in knowing if something like BFT is implemented? If yes, then how and what is the details of the implementation.
Ammu (Tue, 23 Jan 2018 06:06:45 GMT):
front end application what are the things we need to use for hyper ledger fabrics
?
ankitkamra (Tue, 23 Jan 2018 08:03:01 GMT):
Has joined the channel.
yacovm (Tue, 23 Jan 2018 10:34:59 GMT):
@anishman there is no such thing
C0rWin (Tue, 23 Jan 2018 12:12:18 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=68JmpJ3SXHd3DitNq) @ArnabChatterjee the anti entropy in context of gossip is mechanism to take care to bring partitioned nodes or newly connected nodes up to speed by taking care to replicate missing blocks. There is no consensus running at gossip level, there is a layer which leverages natural order of blocks sequence numbers given from ordering service to maintain same order while forwarding block for commit into the ledger. There is block verification in place to check the integrity, hence not really sure why you mention BFT in this context.
C0rWin (Tue, 23 Jan 2018 12:13:09 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=vXrDaFgFGJcxfDXLs) @Ammu could you please forward your question to #fabric channel as not strictly related to gossip component
ArnabChatterjee (Tue, 23 Jan 2018 14:05:07 GMT):
@C0rWin - Thank you so much for your inputs. Any further hints, what this block verification covers? Do you mean VSCC?
C0rWin (Tue, 23 Jan 2018 14:06:21 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=9PxkZxhaJXJ8ZWtxy) @ArnabChatterjee no, I mean signatures of the ordering service, e.g. we do not want to disseminate forged block
ArnabChatterjee (Tue, 23 Jan 2018 14:07:26 GMT):
thank you so much.
ArnabChatterjee (Tue, 23 Jan 2018 14:08:02 GMT):
@yacovm - Hi, I was looking at https://www.youtube.com/watch?v=qpFzZRNKKp0. Here, What do you mean by Consensus node? Leading peer?
yacovm (Tue, 23 Jan 2018 14:17:36 GMT):
ordering node
yacovm (Tue, 23 Jan 2018 14:17:42 GMT):
wow it's a very old demo
ArnabChatterjee (Tue, 23 Jan 2018 23:37:17 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=nT75CLCza2Pg8jGdT) @yacovm You don't think its relevant any more?
StevenXu (Wed, 24 Jan 2018 01:20:46 GMT):
Has joined the channel.
anishman (Wed, 24 Jan 2018 05:34:35 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=J8m7YHDBfn3YsCXj2) @yacovm thanks a lot for the reply.
Ammu (Wed, 24 Jan 2018 10:54:51 GMT):
in couchdb how to set primary key and foreign keys?
yacovm (Wed, 24 Jan 2018 11:01:21 GMT):
@Ammu please ask this in #fabric-ledger
novusopt (Wed, 24 Jan 2018 17:12:02 GMT):
Has joined the channel.
novusopt (Wed, 24 Jan 2018 17:12:52 GMT):
Hi, I am observing following log messages
```2018-01-24 17:10:32.835 UTC [gossip/comm] createConnection -> WARN 2d00 Remote endpoint claims to be a different peer, expected [106 142 75 11 126 62 160 113 178 193 205 134 222 109 14 241 56 244 244 246 42 48 218 9 59 173 66 106 109 46 93 92] but got [44 203 37 178 247 63 230 106 164 47 187 81 109 70 172 249 216 208 152 177 57 140 15 117 134 155 170 140 30 37 238 131]
2018-01-24 17:10:32.835 UTC [gossip/comm] sendToEndpoint -> WARN 2d01 Failed obtaining connection for 0.0.0.0:7051, PKIid:[106 142 75 11 126 62 160 113 178 193 205 134 222 109 14 241 56 244 244 246 42 48 218 9 59 173 66 106 109 46 93 92] reason: Authentication failure````
novusopt (Wed, 24 Jan 2018 17:13:07 GMT):
can someone explain what does this mean?
novusopt (Wed, 24 Jan 2018 17:13:53 GMT):
```2018-01-24 17:13:12.862 UTC [gossip/discovery] expireDeadMembers -> WARN 3be6 Entering [[254 218 99 74 23 130 124 238 222 77 20 100 40 102 250 235 86 132 245 53 245 187 249 17 21 200 231 116 1 117 94 24]]
2018-01-24 17:13:12.862 UTC [gossip/discovery] expireDeadMembers -> WARN 3be7 Closing connection to Endpoint: peer2.org1.dbg-ledger.de:7051, InternalEndpoint: 0.0.0.0:7051, PKI-ID: [254 218 99 74 23 130 124 238 222 77 20 100 40 102 250 235 86 132 245 53 245 187 249 17 21 200 231 116 1 117 94 24], Metadata: []
2018-01-24 17:13:12.862 UTC [gossip/discovery] expireDeadMembers -> WARN 3be8 Exiting```
novusopt (Wed, 24 Jan 2018 17:13:53 GMT):
```2018-01-24 17:13:12.862 UTC [gossip/discovery] expireDeadMembers -> WARN 3be6 Entering [[254 218 99 74 23 130 124 238 222 77 20 100 40 102 250 235 86 132 245 53 245 187 249 17 21 200 231 116 1 117 94 24]]
2018-01-24 17:13:12.862 UTC [gossip/discovery] expireDeadMembers -> WARN 3be7 Closing connection to Endpoint: peer2.org1.my-ledger.com:7051, InternalEndpoint: 0.0.0.0:7051, PKI-ID: [254 218 99 74 23 130 124 238 222 77 20 100 40 102 250 235 86 132 245 53 245 187 249 17 21 200 231 116 1 117 94 24], Metadata: []
2018-01-24 17:13:12.862 UTC [gossip/discovery] expireDeadMembers -> WARN 3be8 Exiting```
yacovm (Wed, 24 Jan 2018 17:15:08 GMT):
that's a misconfiguration
yacovm (Wed, 24 Jan 2018 17:15:26 GMT):
most chances are - you used the same certificate for 2 peers
toddinpal (Wed, 24 Jan 2018 20:40:19 GMT):
@yacovm Does the gossip protocol not allow multiple peers to share a certificate?
yacovm (Wed, 24 Jan 2018 20:40:27 GMT):
of course not
toddinpal (Wed, 24 Jan 2018 20:40:40 GMT):
why?
yacovm (Wed, 24 Jan 2018 20:40:51 GMT):
because a certificate is an identity of a node
toddinpal (Wed, 24 Jan 2018 20:41:50 GMT):
and if I want to have copies of that node for HA?
toddinpal (Wed, 24 Jan 2018 20:42:07 GMT):
The nodes have unique IP addresses
yacovm (Wed, 24 Jan 2018 20:45:05 GMT):
don't do that
yacovm (Wed, 24 Jan 2018 20:45:14 GMT):
don't use your own HA methods
yacovm (Wed, 24 Jan 2018 20:45:22 GMT):
if you want HA just have more peers with different certificates
yacovm (Wed, 24 Jan 2018 20:45:29 GMT):
really, fabric does HA for you so you won't have to
toddinpal (Wed, 24 Jan 2018 20:52:51 GMT):
yet for an OSN, we need to do our own HA as the SDKs only support specifying a single url for the ordering service
yacovm (Wed, 24 Jan 2018 20:53:25 GMT):
oh, they do?
yacovm (Wed, 24 Jan 2018 20:53:28 GMT):
:/
toddinpal (Wed, 24 Jan 2018 20:54:14 GMT):
at least the node.js SDK you specify a single url for the orderer...
toddinpal (Wed, 24 Jan 2018 20:54:53 GMT):
no list/array of urls... so we can hide multiple OSNs behind a load balancer...
yacovm (Wed, 24 Jan 2018 20:56:04 GMT):
yes, you can
yacovm (Wed, 24 Jan 2018 20:56:14 GMT):
make it a network-level LB
toddinpal (Wed, 24 Jan 2018 20:56:21 GMT):
right...
toddinpal (Wed, 24 Jan 2018 20:56:29 GMT):
have to think some more about peers though
novusopt (Thu, 25 Jan 2018 06:35:52 GMT):
@yacovm thanks for the response I will check. One more question what is the difference between CORE_PEER_GOSSIP_EXTERNALENDPOINT and CORE_PEER_GOSSIP_BOOTSTRAP?
novusopt (Thu, 25 Jan 2018 07:12:31 GMT):
I have checked and each peer has a different certificate...
yacovm (Thu, 25 Jan 2018 08:29:00 GMT):
Bootstrap is a peer it contacts when in starts up in order to know other peers im the network
yacovm (Thu, 25 Jan 2018 08:29:22 GMT):
External endpoont is the address you publish to peers in other organizations
novusopt (Thu, 25 Jan 2018 14:37:19 GMT):
@yacovm thx
novusopt (Thu, 25 Jan 2018 14:41:28 GMT):
I found my issue
novusopt (Thu, 25 Jan 2018 14:41:57 GMT):
I didnt set CORE_PEER_GOSSIP_ENDPOINT
niyuelin (Fri, 26 Jan 2018 08:49:20 GMT):
Has joined the channel.
guoger (Wed, 31 Jan 2018 03:28:49 GMT):
Has joined the channel.
guoger (Wed, 31 Jan 2018 03:31:07 GMT):
a quick question, how is leander election done in Fabric if not statically configured?
guoger (Wed, 31 Jan 2018 03:33:54 GMT):
never mind, found in code :P
Ammu (Wed, 31 Jan 2018 07:34:08 GMT):
where our data's will store in couchdb?
C0rWin (Wed, 31 Jan 2018 08:05:48 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=ekiaRFE78jTJCX2X4) @Ammu please forward your question to #fabric-ledger
Dark_Knight (Wed, 31 Jan 2018 13:57:07 GMT):
Has joined the channel.
Dark_Knight (Wed, 31 Jan 2018 13:57:13 GMT):
Hello everyone, I am working on Hyperledger Fabric Consensus network. I wanted to know if it is possible to implement the rollback mechanism in case of transaction failure and if yes, which type of Consensus has to be employed. Or if some other properties have also to be taken into consideration?
yacovm (Wed, 31 Jan 2018 14:04:07 GMT):
please forward this question to #fabric-orderer
frankz (Thu, 01 Feb 2018 03:11:30 GMT):
Has joined the channel.
MartinKrmer (Thu, 01 Feb 2018 10:17:33 GMT):
Has joined the channel.
MartinKrmer (Thu, 01 Feb 2018 10:34:31 GMT):
Hey, is this channel alive?
yacovm (Thu, 01 Feb 2018 10:38:36 GMT):
of course!
C0rWin (Thu, 01 Feb 2018 11:38:01 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=RHe8DiLKDSKg9XJkP) @MartinKrmer ping
MartinKrmer (Thu, 01 Feb 2018 11:38:15 GMT):
thumbnail_image001.jpg
C0rWin (Thu, 01 Feb 2018 11:39:35 GMT):
no sure I'm following your question
C0rWin (Thu, 01 Feb 2018 11:39:50 GMT):
what is so specific there to gossip in your schema?
MartinKrmer (Thu, 01 Feb 2018 11:40:05 GMT):
The modelling is a little bit freestyle, but the right part should be gossip
C0rWin (Thu, 01 Feb 2018 11:40:14 GMT):
ah?
MartinKrmer (Thu, 01 Feb 2018 11:40:30 GMT):
"Peer Communication Fabric"
C0rWin (Thu, 01 Feb 2018 11:40:40 GMT):
gossip intend to be used to distribute blocks across the peers
C0rWin (Thu, 01 Feb 2018 11:40:57 GMT):
of course scalable
MartinKrmer (Thu, 01 Feb 2018 11:41:52 GMT):
Yes I know but is it feasible to theoretically apply gossip for the peers: patient + all of his healthcare provider
MartinKrmer (Thu, 01 Feb 2018 11:42:22 GMT):
in such a schema =(
C0rWin (Thu, 01 Feb 2018 11:43:46 GMT):
apply to do what?
MartinKrmer (Thu, 01 Feb 2018 11:50:51 GMT):
use case
Vadim (Thu, 01 Feb 2018 11:52:08 GMT):
@MartinKrmer is it possible to apply tcp/ip to email sending?
MartinKrmer (Thu, 01 Feb 2018 11:56:51 GMT):
Did not yet think about it, but theoretically yes, would it make more feasible?
MartinKrmer (Thu, 01 Feb 2018 11:58:38 GMT):
Besides; do the peers need to be online both for using gossip tranmission?
C0rWin (Thu, 01 Feb 2018 11:59:17 GMT):
I think I've lost the context
C0rWin (Thu, 01 Feb 2018 12:00:16 GMT):
how this is all connected together? gossip, email sending, tcp... :/
C0rWin (Thu, 01 Feb 2018 12:00:42 GMT):
as a matter of fact, gossip runs over tcp/ip
Vadim (Thu, 01 Feb 2018 12:00:48 GMT):
@C0rWin I was trying to show him that his question makes little sense in that form
C0rWin (Thu, 01 Feb 2018 12:02:09 GMT):
> Besides; do the peers need to be online both for using gossip tranmission?
peer has to be online and connected to accept the communication and being able to receive message
C0rWin (Thu, 01 Feb 2018 12:02:30 GMT):
there is a way to sync up off lines nodes, not directly connected to gossip though
C0rWin (Thu, 01 Feb 2018 12:02:51 GMT):
also, there is no such thing as submitting peer
C0rWin (Thu, 01 Feb 2018 12:03:07 GMT):
I have to admit your diagram makes very little sense to me
C0rWin (Thu, 01 Feb 2018 12:04:37 GMT):
even less, speaking of gossip context
C0rWin (Thu, 01 Feb 2018 12:04:50 GMT):
what exactly you are trying to achieve?
MartinKrmer (Thu, 01 Feb 2018 12:07:00 GMT):
Sorry I may expressed myself wrong; I would like to model a architecture that enables all healthcare provider to share information with the patient as well as help the healthcare provider to synchronize their data
MartinKrmer (Thu, 01 Feb 2018 12:10:13 GMT):
(with healthcare provider I mean the physician etc where the patient has "account")
MartinKrmer (Thu, 01 Feb 2018 12:10:59 GMT):
I may messed-up gossip with https://github.com/hyperledger-archives/fabric/wiki/Next-Consensus-Architecture-Proposal -> submitting node
shailaja.mahara (Wed, 07 Feb 2018 05:58:16 GMT):
Has joined the channel.
shailaja.mahara (Thu, 08 Feb 2018 05:12:59 GMT):
m working on CentOS and even after installing golang, I am getting an error while running the command: go get -u --tags nopkcs11 github.com/hyperledger/fabric/core/chaincode/shim
The error is: bash: go: command not found...
i'm trying to compile the chaincode, following the official docs guidelines.
I followed the the docs to install Go as well
export GOPATH=$HOME/go
export PATH=$PATH:$GOPATH/bin
volkanbaran (Fri, 09 Feb 2018 09:00:05 GMT):
Has joined the channel.
ohmeraka (Mon, 12 Feb 2018 16:22:57 GMT):
Has joined the channel.
Ryan2 (Tue, 13 Feb 2018 14:45:37 GMT):
Has joined the channel.
vudathasaiomkar (Wed, 14 Feb 2018 07:12:53 GMT):
Has joined the channel.
niteshsolanki (Thu, 15 Feb 2018 04:02:38 GMT):
If an org leader fails to send correct blocks to the rest of the peers in an org, how would the rest of the nodes slecet a next source to get valid blocks ?
niteshsolanki (Thu, 15 Feb 2018 04:02:38 GMT):
If an org leader fails to send correct blocks to the rest of the peers in an org, how would the rest of the nodes select the next source to get valid blocks ?
C0rWin (Thu, 15 Feb 2018 22:40:07 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=JfXP38hLFufFbBHGW) @niteshsolanki if there is network partition, that means other nodes won't see a leader for a while, hence will re-elect new leader. otherwise three is a pull mechanism which help other nodes in sync
eclairamb (Sat, 17 Feb 2018 17:59:05 GMT):
Has joined the channel.
vitiko (Mon, 19 Feb 2018 11:42:40 GMT):
Has joined the channel.
sampath06 (Thu, 22 Feb 2018 14:16:59 GMT):
I am trying to setup up peers on different servers. How do they discover each other? Any bootstrapping needs to be done?
yacovm (Thu, 22 Feb 2018 14:17:07 GMT):
yes
yacovm (Thu, 22 Feb 2018 14:17:09 GMT):
@sampath06
yacovm (Thu, 22 Feb 2018 14:17:23 GMT):
You have ways in which peers discover each other
yacovm (Thu, 22 Feb 2018 14:17:38 GMT):
peers that are in the same org, do that via the gossip.bootstrap parameter
yacovm (Thu, 22 Feb 2018 14:17:47 GMT):
peers of different orgs, discover each other via anchor peers
sampath06 (Thu, 22 Feb 2018 14:20:12 GMT):
So I set the anchor peer to peer0 and am trying to bring up peer1 on another server. Peer0 comes up and joins the channel correctly. But when trying to bring peer1 up, I am getting the following error
```
peer1.fi1.com | 2018-02-22 14:17:54.330 UTC [gossip/discovery] func1 -> WARN 1df Could not connect to {127.0.0.1:7051 [] [] 127.0.0.1:7051
yacovm (Thu, 22 Feb 2018 14:20:33 GMT):
aha, very good
sampath06 (Thu, 22 Feb 2018 14:20:55 GMT):
configtx.yaml has
```
AnchorPeers:
# AnchorPeers defines the location of peers which can be used
# for cross org gossip communication. Note, this value is only
# encoded in the genesis block in the Application section context
- Host: peer0.fi1.com
Port: 7051
```
yacovm (Thu, 22 Feb 2018 14:20:58 GMT):
it has nothing to do with anchor peers though
yacovm (Thu, 22 Feb 2018 14:21:07 GMT):
that message is when it tries to connect to the bootstrap peer
yacovm (Thu, 22 Feb 2018 14:21:15 GMT):
that is in `core.yaml`
yacovm (Thu, 22 Feb 2018 14:21:29 GMT):
https://github.com/hyperledger/fabric/blob/master/sampleconfig/core.yaml#L136
yacovm (Thu, 22 Feb 2018 14:21:42 GMT):
what version are you running?
yacovm (Thu, 22 Feb 2018 14:22:14 GMT):
are you running with docker?
sampath06 (Thu, 22 Feb 2018 14:23:29 GMT):
Yes.. using latest
```
hyperledger/fabric-peer latest f00c5d490d19 3 weeks ago 170MB
hyperledger/fabric-peer x86_64-1.1.0-alpha f00c5d490d19 3 weeks ago 170MB
```
sampath06 (Thu, 22 Feb 2018 14:23:43 GMT):
So how do I change it when the peer0 is on a different server?
yacovm (Thu, 22 Feb 2018 14:23:49 GMT):
ok then what is the value of `CORE_PEER_GOSSIP_BOOTSTRAP` ?
yacovm (Thu, 22 Feb 2018 14:24:06 GMT):
are you using docker-compose?
sampath06 (Thu, 22 Feb 2018 14:25:58 GMT):
yes. havent set the value. Let me try setting that to peer0 and see if that works.
sampath06 (Thu, 22 Feb 2018 14:36:52 GMT):
@yacovm Am getting the following error now
```
peer1.fi1.com | 2018-02-22 14:35:11.482 UTC [gossip/discovery] func1 -> WARN 1df Could not connect to {peer0.fi1.com:7051 [] [] peer0.fi1.com:7051
yacovm (Thu, 22 Feb 2018 14:38:13 GMT):
1) first try to log into the container and see if the address is resolved
2) Try to see if the port is reachable
3) Set `CORE_LOGGING_GRPC=debug` in the environment variables, it might be a TLS mis-configuration
sampath06 (Thu, 22 Feb 2018 14:45:46 GMT):
@yacovm Thank you so much. Fixed the issue. The port was not open in Azure where it is hosted.
yacovm (Thu, 22 Feb 2018 14:45:58 GMT):
no prob, happy to help :)
david_dornseifer (Thu, 22 Feb 2018 15:41:21 GMT):
Hi, does somebody know how to approach this problem? ```[gossip/discovery] func1 -> WARN 031 Could not connect to {peer0.org1:7051 [] [] peer0.org1} : x509: certificate signed by unknown authority```
david_dornseifer (Thu, 22 Feb 2018 15:41:21 GMT):
Hi, does somebody know how to approach this problem? ```[gossip/discovery] func1 -> WARN 031 Could not connect to {peer0.org1:7051 [] [] peer0.org1:7051} : x509: certificate signed by unknown authority```
david_dornseifer (Thu, 22 Feb 2018 15:43:54 GMT):
a similar one: ```[gossip/gossip] func1 -> WARN 030 Deep probe of peer0.org2:7051 failed: x509: certificate signed by unknown authority```
yacovm (Thu, 22 Feb 2018 15:55:58 GMT):
@david_dornseifer I'd guess that's a misconfiguration issue
david_dornseifer (Thu, 22 Feb 2018 15:57:13 GMT):
@yacovm hmm think the x509 cert for the orgs should be distributed via the channel config right? - the tls section
yacovm (Thu, 22 Feb 2018 15:57:22 GMT):
correct
yacovm (Thu, 22 Feb 2018 15:57:44 GMT):
unless its in the same org, which is not the case here
david_dornseifer (Thu, 22 Feb 2018 15:59:17 GMT):
hmm ok, already checked it but i'll have another look :) - there is no no need to distribute the tls root cas to the peers manually right?
yacovm (Thu, 22 Feb 2018 15:59:58 GMT):
you only need the root cert of the peer's org to be present in `tls/ca.cert`
david_dornseifer (Thu, 22 Feb 2018 16:00:33 GMT):
thx - will check the config again
Wangrj (Fri, 23 Feb 2018 00:56:20 GMT):
Has joined the channel.
SriramaSharma (Fri, 23 Feb 2018 03:46:37 GMT):
Has joined the channel.
SashiKanth (Fri, 23 Feb 2018 08:08:05 GMT):
Has joined the channel.
SashiKanth (Fri, 23 Feb 2018 08:20:09 GMT):
Hi,
Inorder to able to have a setup of different peers in different servers,
do i need to have docker swarm and have a overlay network setup
yacovm (Fri, 23 Feb 2018 09:24:25 GMT):
no
SashiKanth (Sat, 24 Feb 2018 08:05:03 GMT):
Hi,
I got this error when connecting a peer to the already exsiting fabric network, but the peer resides in different host
Failed to dial 10.0.0.6:7050: connection error: desc = "transport: authentication handshake failed: x509: cannot validate certificate for 10.0.0.6 because it doesn't contain any IP SANs"; please retry.
sampath06 (Sun, 25 Feb 2018 16:21:44 GMT):
@yacovm @david_dornseifer I am facing the same issue. Were you able to resolve it. When I try to connect the peers using the CORE_PEER_GOSSIP_BOOTSTRAP variable, I am hitting the following errors
```
sampath06 (Sun, 25 Feb 2018 16:21:44 GMT):
@yacovm @david_dornseifer I am facing the same issue. Were you able to resolve it. When I try to connect the peers using the CORE_PEER_GOSSIP_BOOTSTRAP variable, I am hitting the following errors
```
peer0.fi1.com | 2018-02-25 16:21:31.164 UTC [gossip/discovery] func1 -> WARN 1f6 Could not connect to {peer0.anchor1.com:7051 [] [] peer0.anchor1.com:7051
yacovm (Sun, 25 Feb 2018 16:22:09 GMT):
I don't see anything, it's empty
yacovm (Sun, 25 Feb 2018 16:22:16 GMT):
oh
sampath06 (Sun, 25 Feb 2018 16:22:25 GMT):
Sorry.. Hit enter by mistake. Edited it
yacovm (Sun, 25 Feb 2018 16:22:44 GMT):
...
yacovm (Sun, 25 Feb 2018 16:22:54 GMT):
did you put a bootstrap peer of another organization?
sampath06 (Sun, 25 Feb 2018 16:23:05 GMT):
Yes..
yacovm (Sun, 25 Feb 2018 16:23:09 GMT):
bootstrap peers are *only* peers in your own organization
sampath06 (Sun, 25 Feb 2018 16:23:17 GMT):
Ah.. Ok
yacovm (Sun, 25 Feb 2018 16:23:22 GMT):
actually, even if the handshake would succeed, gossip would reject it :)
yacovm (Sun, 25 Feb 2018 16:23:30 GMT):
claiming it's a foreign organization
sampath06 (Sun, 25 Feb 2018 16:23:47 GMT):
How do the peers of different organisation join the network. Is it just through the channel join?
yacovm (Sun, 25 Feb 2018 16:24:15 GMT):
yes'
sampath06 (Sun, 25 Feb 2018 16:24:51 GMT):
Great. Thanks for the quick help again
yzhivkov (Sun, 25 Feb 2018 18:34:27 GMT):
Has joined the channel.
kapilAtrey (Mon, 26 Feb 2018 09:05:41 GMT):
Has joined the channel.
mflipw (Mon, 26 Feb 2018 11:05:33 GMT):
Has joined the channel.
SashiKanth (Tue, 27 Feb 2018 10:23:04 GMT):
Hi,
I got this error when connecting a peer to the already exsiting fabric network, but the peer resides in different host
Failed to dial 10.0.0.6:7050: connection error: desc = "transport: authentication handshake failed: x509: cannot validate certificate for 10.0.0.6 because it doesn't contain any IP SANs"; please retry.
yacovm (Tue, 27 Feb 2018 10:24:30 GMT):
you can use a cryptogen of version v1.1
yacovm (Tue, 27 Feb 2018 10:24:38 GMT):
and then it will add IP SANs to the certificates
yacovm (Tue, 27 Feb 2018 10:24:40 GMT):
@SashiKanth
SashiKanth (Tue, 27 Feb 2018 10:44:14 GMT):
where can i get cryptogen of version v1.1 ..... I cant find it ...
SashiKanth (Tue, 27 Feb 2018 10:44:50 GMT):
where can i get cryptogen of version v1.1 ..... I cant find it ...
@yacovm
yacovm (Tue, 27 Feb 2018 10:45:41 GMT):
clone https://github.com/hyperledger/fabric.git and switch to the master branch
yacovm (Tue, 27 Feb 2018 10:45:43 GMT):
and compile it
yacovm (Tue, 27 Feb 2018 10:45:54 GMT):
via: `make cryptogen`
SashiKanth (Tue, 27 Feb 2018 10:46:28 GMT):
ok thanks
SashiKanth (Tue, 27 Feb 2018 11:03:34 GMT):
@yacovm
Is there any way that i can generate the certs without cryptogen
yacovm (Tue, 27 Feb 2018 11:04:09 GMT):
yes with openssl
SashiKanth (Tue, 27 Feb 2018 11:04:39 GMT):
by providing the same crypto-config.yaml file
SashiKanth (Tue, 27 Feb 2018 11:04:42 GMT):
??
Ammu (Tue, 27 Feb 2018 11:15:44 GMT):
stsrt.png
Ammu (Tue, 27 Feb 2018 11:15:49 GMT):
can any1 solve this error?
yacovm (Tue, 27 Feb 2018 11:18:11 GMT):
this isn't related to this channel @Ammu
SashiKanth (Tue, 27 Feb 2018 11:28:01 GMT):
@yacovm
SashiKanth (Tue, 27 Feb 2018 11:28:29 GMT):
curl -sSL https://goo.gl/6wtTN5 | bash -s 1.1.0-alpha
still it has got cryptogen v 1.0.5
but not v 1.1
yacovm (Tue, 27 Feb 2018 11:28:38 GMT):
uh oh...
yacovm (Tue, 27 Feb 2018 11:28:45 GMT):
@mastersingh24 ^
mastersingh24 (Tue, 27 Feb 2018 15:54:23 GMT):
I just ran the download script and I get 1.1.0-alpha binaries ...
mastersingh24 (Tue, 27 Feb 2018 15:55:15 GMT):
```
Garis-MacBook-Pro:fabric-samples gsingh$ ./bin/cryptogen version
cryptogen:
Version: 1.1.0-alpha
Go version: go1.9
OS/Arch: darwin/amd64
```
chandrakanthm (Wed, 28 Feb 2018 09:54:44 GMT):
Has joined the channel.
dampuero (Wed, 28 Feb 2018 15:06:44 GMT):
Has joined the channel.
joaquimpedrooliveira (Thu, 01 Mar 2018 13:00:35 GMT):
Has joined the channel.
joaquimpedrooliveira (Thu, 01 Mar 2018 13:02:44 GMT):
Hello, all. I have a question regarding gossip configuration in a network with multiple peers from multiple orgs: what happens if I don't configure the env variables `CORE_PEER_GOSSIP_*` in peer config?
yacovm (Thu, 01 Mar 2018 13:05:36 GMT):
it uses the core.yamll values
yacovm (Thu, 01 Mar 2018 13:05:40 GMT):
from the `core.yaml` file
joaquimpedrooliveira (Thu, 01 Mar 2018 13:08:45 GMT):
So in this case all peers will be configured as "org leaders" and receive blocks directly from the orderer service, is this correct?
yacovm (Thu, 01 Mar 2018 13:10:24 GMT):
yes
yacovm (Thu, 01 Mar 2018 13:10:26 GMT):
I think
joaquimpedrooliveira (Thu, 01 Mar 2018 13:12:19 GMT):
In this scenario, what happens to a newly connected peer? Does it receive the existing blocks from the orderer?
joaquimpedrooliveira (Thu, 01 Mar 2018 13:18:22 GMT):
Another question: configuring the env variables `CORE_PEER_GOSSIP_*` is considered *mandatory* or *optional* in a network with many orgs and many peers?
yacovm (Thu, 01 Mar 2018 13:28:17 GMT):
depends on what you want
joaquimpedrooliveira (Thu, 01 Mar 2018 13:34:10 GMT):
please elaborate. I'm trying to understand how to properly configure a network but really don't know the implications, even after reading the docs (fabric.readthedocs.io/en/release/gossip.html)
joaquimpedrooliveira (Thu, 01 Mar 2018 13:34:31 GMT):
I appreciate your help :)
yacovm (Thu, 01 Mar 2018 13:35:35 GMT):
so, you need to define a bootstrap list for your peers for your org
yacovm (Thu, 01 Mar 2018 13:35:43 GMT):
ummm... and also define anchor peers in the configtx.yaml
yacovm (Thu, 01 Mar 2018 13:35:53 GMT):
I think that's basically it, the rest should work out of the box
joaquimpedrooliveira (Thu, 01 Mar 2018 13:39:37 GMT):
I understand these are the steps needed to properly configure gossip, right?
joaquimpedrooliveira (Thu, 01 Mar 2018 13:39:49 GMT):
That's ok to me
joaquimpedrooliveira (Thu, 01 Mar 2018 13:40:24 GMT):
What I trying to figure out is what are the impacts of not configuring the gossip and use the default config
yacovm (Thu, 01 Mar 2018 13:41:08 GMT):
ah - all peers will connect to the orderer
yacovm (Thu, 01 Mar 2018 13:41:11 GMT):
that's it basically
joaquimpedrooliveira (Thu, 01 Mar 2018 13:43:11 GMT):
Ok...but do peer discovery and channel membership, ledger dissemination and update ledger data for newly connected peers will work fine?
joaquimpedrooliveira (Thu, 01 Mar 2018 13:43:11 GMT):
Ok...but will peer discovery and channel membership, ledger dissemination and update ledger data for newly connected peers work fine?
joaquimpedrooliveira (Thu, 01 Mar 2018 13:44:06 GMT):
I'm assuming it will, but the messages will be receveid from the orderer instead of from other peers, but I want to be sure I'm understanding correctly. :)
joaquimpedrooliveira (Thu, 01 Mar 2018 13:47:16 GMT):
And when all peers connected to the orderer starts to impact the network performance?
yacovm (Thu, 01 Mar 2018 13:47:44 GMT):
> Ok...but will peer discovery and channel membership, ledger dissemination and update ledger data for newly connected peers work fine?
it will not
yacovm (Thu, 01 Mar 2018 13:47:56 GMT):
unless you have anchor peers
yacovm (Thu, 01 Mar 2018 13:48:07 GMT):
then it would
yacovm (Thu, 01 Mar 2018 13:48:14 GMT):
but you also need to configure your endpoint
yacovm (Thu, 01 Mar 2018 13:48:23 GMT):
`CORE_PEER_GOSSIP_ENDPOINT`
joaquimpedrooliveira (Thu, 01 Mar 2018 13:55:39 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=xWf8NsJFJPmFrj3zK) @yacovm pointing to the anchor peer?
yacovm (Thu, 01 Mar 2018 14:02:40 GMT):
in the configtx.yaml
yacovm (Thu, 01 Mar 2018 14:02:46 GMT):
sorry, I'm in a call so can't really elaborate
joaquimpedrooliveira (Thu, 01 Mar 2018 14:09:03 GMT):
No problem, I can wait :)
yacovm (Thu, 01 Mar 2018 14:16:37 GMT):
please write concrete questions
yacovm (Thu, 01 Mar 2018 14:16:41 GMT):
and I'll answer when I am back
yacovm (Thu, 01 Mar 2018 14:16:50 GMT):
@joaquimpedrooliveira
AndrewRy 1 (Thu, 01 Mar 2018 16:29:45 GMT):
Has joined the channel.
joaquimpedrooliveira (Thu, 01 Mar 2018 17:09:36 GMT):
- What is the impact of not configuring the gossip (use the default config)
- If gossip is not enabled but anchor peers are defined in `configtx.yaml` and peer address is configured in `CORE_PEER_GOSSIP_EXTERNALENDPOINT`, do we have the same benefits from configuring gossip?
- When all peers are connected to the orderer (as org leaders), is there any performance degradation due to network size?
- In what situations/scenarios configuring gossip properly in a network is recommended?
joaquimpedrooliveira (Thu, 01 Mar 2018 17:09:36 GMT):
- What is the impact of not configuring the gossip (use the default config)?
- If gossip is not enabled but anchor peers are defined in `configtx.yaml` and peer address is configured in `CORE_PEER_GOSSIP_EXTERNALENDPOINT`, do we have the same benefits from configuring gossip?
- When all peers are connected to the orderer (as org leaders), is there any performance degradation due to network size?
- In what situations/scenarios configuring gossip properly in a network is recommended?
yacovm (Thu, 01 Mar 2018 17:59:45 GMT):
> - What is the impact of not configuring the gossip (use the default config)?
You need anchor peers to know of other peers. If you had `CORE_PEER_GOSSIP_BOOTSTRAP` too then you could know of peers in your own org.
> - If gossip is not enabled but anchor peers are defined in `configtx.yaml` and peer address is configured in `CORE_PEER_GOSSIP_EXTERNALENDPOINT`, do we have the same benefits from configuring gossip?
if you have external endpoint, your peer is visible to other orgs and it is conversing with peers from other orgs. But keep in mind you need to configure `CORE_PEER_ADDRESS` to have gossip with peers in your own org.
> - When all peers are connected to the orderer (as org leaders), is there any performance degradation due to network size?
You will hit a bandwidth bottleneck at the orderer if the network can't keep up, but that's it.
> - In what situations/scenarios configuring gossip properly in a network is recommended?
when you have lots of peers and lots of organizations, and when you use the v1.1 feature called "side DB" - gossip is used in that to pass data between peers.
@joaquimpedrooliveira - ^
joaquimpedrooliveira (Thu, 01 Mar 2018 18:15:36 GMT):
Thank you very much for you help, @yacovm ! :)
yacovm (Thu, 01 Mar 2018 18:16:02 GMT):
np
joaquimpedrooliveira (Thu, 01 Mar 2018 18:44:13 GMT):
one last question, @yacovm: what do you mean by "lots of peers" and "lots of organizations"?
yacovm (Thu, 01 Mar 2018 18:44:30 GMT):
many
joaquimpedrooliveira (Thu, 01 Mar 2018 18:44:41 GMT):
more than 10, 50 or 100? :)
joaquimpedrooliveira (Thu, 01 Mar 2018 18:44:47 GMT):
how big
yacovm (Thu, 01 Mar 2018 18:45:05 GMT):
it depends on the block creation rate and the bandwidth the orderers can give and their amount
yacovm (Thu, 01 Mar 2018 18:45:12 GMT):
just do the math
yacovm (Thu, 01 Mar 2018 18:45:21 GMT):
if you have too many peers pulling too many blocks
yacovm (Thu, 01 Mar 2018 18:45:27 GMT):
and the bandwidth is too low
yacovm (Thu, 01 Mar 2018 18:45:38 GMT):
then you are bottle-necked by bandwidth
joaquimpedrooliveira (Thu, 01 Mar 2018 18:45:59 GMT):
ok, got it! thanks!
joaquimpedrooliveira (Thu, 01 Mar 2018 20:09:42 GMT):
Is it common to see gossip messages like:
```2018-03-01 20:03:26.192 UTC [gossip/channel] handleStateInfSnapshot -> DEBU 3da Channel defaultchannel : Couldn't find org identity of peer �
iSZ�(�� message sent from �ܨ�g]�N��p9 ����Wѭ��4�6
J�����so�ߝ� m�͎
```
joaquimpedrooliveira (Thu, 01 Mar 2018 20:09:42 GMT):
Is it common to see gossip messages like this?
```2018-03-01 20:03:26.192 UTC [gossip/channel] handleStateInfSnapshot -> DEBU 3da Channel defaultchannel : Couldn't find org identity of peer �
iSZ�(�� message sent from �ܨ�g]�N��p9 ����Wѭ��4�6
J�����so�ߝ� m�͎
```
joaquimpedrooliveira (Thu, 01 Mar 2018 20:09:52 GMT):
using Fabric 1.0.6
yacovm (Thu, 01 Mar 2018 21:19:28 GMT):
@joaquimpedrooliveira does it happen constantly or has it stopped after a while?
joaquimpedrooliveira (Fri, 02 Mar 2018 12:23:31 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=ifFkjDqxrvwd4BfbM) @yacovm Stopped after a while.
yacovm (Fri, 02 Mar 2018 12:23:43 GMT):
then it's ok
joaquimpedrooliveira (Fri, 02 Mar 2018 12:29:42 GMT):
thanks!
nirro (Tue, 06 Mar 2018 13:40:21 GMT):
Has joined the channel.
rjones (Tue, 06 Mar 2018 18:12:55 GMT):
dave.enyeart
dave.enyeart (Tue, 06 Mar 2018 20:07:01 GMT):
yacovm
dave.enyeart (Tue, 06 Mar 2018 20:07:07 GMT):
C0rWin
ShereenSallam (Thu, 08 Mar 2018 12:33:11 GMT):
Has joined the channel.
ShereenSallam (Thu, 08 Mar 2018 12:33:14 GMT):
User User_1 added by ShereenSallam.
TobiasN (Mon, 12 Mar 2018 00:14:14 GMT):
Has joined the channel.
MoulaliMvg (Mon, 12 Mar 2018 05:22:11 GMT):
Has joined the channel.
jspark84 (Tue, 13 Mar 2018 13:16:06 GMT):
Has joined the channel.
tuancm (Tue, 13 Mar 2018 16:12:37 GMT):
Has joined the channel.
SashiKanth (Fri, 16 Mar 2018 11:41:14 GMT):
CORE_PEER_GOSSIP_EXTERNALENDPOINT
CORE_PEER_GOSSIP_BOOTSTRAP
why is this env variables for !!!!
?
chenjun-bj (Sat, 17 Mar 2018 01:02:51 GMT):
Has joined the channel.
pankajcheema (Mon, 19 Mar 2018 05:46:14 GMT):
Has joined the channel.
wjzheng (Tue, 20 Mar 2018 19:06:42 GMT):
Has joined the channel.
ShikarSharma (Tue, 20 Mar 2018 22:43:11 GMT):
Has joined the channel.
marksta (Fri, 23 Mar 2018 19:41:48 GMT):
Has left the channel.
Hangyu (Mon, 26 Mar 2018 00:33:21 GMT):
@yacovm I encountered this phenomenon when I tried to measure performance of fabric through multi-thread.
The time spent in block committing phase were huge, nearly taking up 90% of the total.
here is the logging message embedded in the source
```
2018/03/23 14:40:40.858 [INFO ] ★START SEND OS REQUESTID=0.3381804038376005 f822028580ce3e5aaf46ba45446cef535e3022b48e32a19febcfd3283983d5dc
2018/03/23 14:40:40.863 [INFO ] ★END SEND OS REQUESTID=0.3381804038376005 f822028580ce3e5aaf46ba45446cef535e3022b48e32a19febcfd3283983d5dc
2018/03/23 14:40:42.316 [INFO ] ★BLOCK COMMIT REQUESTID=0.3381804038376005 f822028580ce3e5aaf46ba45446cef535e3022b48e32a19febcfd3283983d5dc
```
from which you can see that it took 1449ms in the blocking committing.
I was wondering what are causing this delay.
My guess would be something happened in one of the goroutine waiting for request? Or the influences from gossip?
Hangyu (Mon, 26 Mar 2018 00:44:01 GMT):
by the way, I was using fabric1.0.0
C0rWin (Mon, 26 Mar 2018 12:11:46 GMT):
@Hangyu how mane peers did you have while doing your measurements? Which state DB engine have you used? What was the endorsement policy? How many transactions do you have in the block? What was the average throughput?
C0rWin (Mon, 26 Mar 2018 12:15:18 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=X8xdG5tXWYk8HiKe4) @SashiKanth
`CORE_PEER_GOSSIP_EXTERNALENDPOINT ` - this is the endpoint peer will advertise to make peers from other organizations to get connected to it (needed if your peers behind a NAT for instance)
`CORE_PEER_GOSSIP_BOOTSTRAP ` - this is the set of peers to be used to build the initial membership
yoshihara.y (Tue, 27 Mar 2018 00:36:20 GMT):
Has joined the channel.
Hangyu (Tue, 27 Mar 2018 02:26:28 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=2wpimxN5jbg5GNFZ9) @C0rWin My apologies for the incomplete information.
peer number: 4 peers from 4 organizations
state DB engine: levelDB
endorsement policy: require signatures from all organizations
transaction number in a block: 10
average throughout: 100 TPS
thakurnikk (Tue, 27 Mar 2018 06:31:07 GMT):
Has joined the channel.
ruffsl (Tue, 27 Mar 2018 16:20:54 GMT):
Has joined the channel.
qingsongGuo (Fri, 30 Mar 2018 07:22:49 GMT):
Has joined the channel.
ericmvaughn (Fri, 30 Mar 2018 22:32:46 GMT):
Has joined the channel.
JiuZhuYou (Sat, 31 Mar 2018 10:39:46 GMT):
Has joined the channel.
jarvis26 (Sat, 31 Mar 2018 15:51:19 GMT):
Has joined the channel.
jarvis26 (Sat, 31 Mar 2018 15:53:26 GMT):
Hi.. I am using `CORE_PEER_GOSSIP_ORGLEADER and CORE_PEER_GOSSIP_USELEADERELECTION` settings in my fabric network which consists of 3 organizations with 3 peer each. I have set `CORE_PEER_GOSSIP_ORGLEADER=true` for one peer in each organization and false for the others and `CORE_PEER_GOSSIP_USELEADERELECTION=false` for all the peers. Now, only my leaders are getting the blocks delivered and all the non leaders aren't. Could anybody please help me on this. Thanks.
yacovm (Sat, 31 Mar 2018 19:58:43 GMT):
@jarvis26 looks like you don't have bootstrap peers or anchor peers, and the non leaders don't know about the leader peers and vice versa.
MeenakshiSingh (Sun, 01 Apr 2018 05:57:17 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=chSbafbjJRsg8tSEL) @yacovm ok..so, ideally what should be given in bootstrap peers, other peers of the same organization or they can belong to different organizations. Also, how and what do I set the anchor peers to?
yacovm (Sun, 01 Apr 2018 07:50:41 GMT):
@MeenakshiSingh - the same org only. Read the comments in the core.yaml.
yacovm (Sun, 01 Apr 2018 07:50:47 GMT):
you set anchor peers via a config update
MeenakshiSingh (Sun, 01 Apr 2018 08:02:53 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=iKDT5TMqvZR8N2p98) @yacovm ok..so while creating the channel genesis, I created the org anchor peers using the configtxgen tool. This gave me channel anchor peer artifacts. Now, what do I use these for?
MeenakshiSingh (Sun, 01 Apr 2018 08:02:53 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=iKDT5TMqvZR8N2p98) @yacovm ok..so while creating the channel genesis, I created the org anchor peers using the configtxgen tool. This gave me channel anchor peer artifacts. Now, what do I use these for? `Anchor peers - used to advertise peers from different organization to eventually build up shared membership view of all peers in the channels from all organizations.` How does this happen?
yacovm (Sun, 01 Apr 2018 08:37:29 GMT):
not sure what you're asking
yacovm (Sun, 01 Apr 2018 08:37:41 GMT):
but you need to submit the anchor peer transaction to the orderer
MeenakshiSingh (Sun, 01 Apr 2018 08:55:05 GMT):
I executed the following commands for generating the channel and anchor peer artifacts:
MeenakshiSingh (Sun, 01 Apr 2018 08:55:11 GMT):
```CONFIGTXGEN=/opt/gopath/src/github.com/hyperledger/fabric/build/bin/configtxgen
$CONFIGTXGEN -profile TwoOrgsOrdererGenesis -outputBlock genesis.block
$CONFIGTXGEN -profile TwoOrgsChannel -outputCreateChannelTx channel1.tx -channelID channel1
$CONFIGTXGEN -profile TwoOrgsChannel -outputAnchorPeersUpdate Org1MSPanchors.tx -channelID channel1 -asOrg Org1MSP
$CONFIGTXGEN -profile TwoOrgsChannel -outputAnchorPeersUpdate Org2MSPanchors.tx -channelID channel1 -asOrg Org2MSP```
MeenakshiSingh (Sun, 01 Apr 2018 08:56:53 GMT):
Not sure where these `Org1MSPanchors.tx` and `Org2MSPanchors.tx` would be used as I am able to create and transact upon a channel which I create using the `channel1.tx` artifact.
yacovm (Sun, 01 Apr 2018 09:04:36 GMT):
look at the e2e_cli in examples folder
yacovm (Sun, 01 Apr 2018 09:04:42 GMT):
specifically in script/script.sh
rsherwood (Mon, 02 Apr 2018 14:49:56 GMT):
How do a I work out how many signing operations there will be for peer / client / orders (non tls signing that is) in steady state ? I need this to make sure that our HSMs have sufficient capacity. I really want the answer for fabric 1.0 but it would be useful to know if this would change with 1.1.
Its clear that the submitter / endorsers will each sign for a transaction, as will the orderers each sign once per block.
I've read that the gossip protocol also uses signed blocks for "keep alive".
Are there "keep alive" between peers and orderer ? Is that per channel ? Does it sign again each message ?
If we put anchor peers into the config how does that add to the signing?
If we have multiple peers in an organisation how does that affect the signing ?
yacovm (Mon, 02 Apr 2018 14:52:37 GMT):
signed blocks for keep alives?
yacovm (Mon, 02 Apr 2018 14:52:58 GMT):
we send signed "alive messages" for keep alives
yacovm (Mon, 02 Apr 2018 14:53:09 GMT):
the keep alive between peers and orderers are at the grpc level
rsherwood (Mon, 02 Apr 2018 15:46:22 GMT):
Does the peer sign those blocks ?
rsherwood (Mon, 02 Apr 2018 15:46:22 GMT):
Does the peer sign those blocks ? Read the blocks says "Online peers indicate their availability by continually broadcasting “alive” messages, with each containing the public key infrastructure (PKI) ID and the *signature* of the sender over the message."
"Although TLS certs are also used, it is the peer certificates that are authenticated in the gossip layer. "
rsherwood (Mon, 02 Apr 2018 15:46:22 GMT):
Does the peer sign those blocks ? Read the docs says "Online peers indicate their availability by continually broadcasting “alive” messages, with each containing the public key infrastructure (PKI) ID and the *signature* of the sender over the message."
"Although TLS certs are also used, it is the peer certificates that are authenticated in the gossip layer. "
yacovm (Mon, 02 Apr 2018 16:09:08 GMT):
peers don't sign blocks
yacovm (Mon, 02 Apr 2018 16:10:27 GMT):
peers broadcast their ledger height
yacovm (Mon, 02 Apr 2018 16:10:29 GMT):
to other peers
rsherwood (Mon, 02 Apr 2018 16:18:14 GMT):
Sorry did not mean blocks . Do peers sign keep alive massages ? So do they sign these ledger heights ?
rsherwood (Mon, 02 Apr 2018 16:18:14 GMT):
Sorry did not mean blocks . Do peers sign keep alive massages ? So do they sign these ledger heights ? If they do how often do they resign. Every time they broadcast >
rsherwood (Mon, 02 Apr 2018 16:18:14 GMT):
Sorry did not mean blocks . Do peers sign keep alive massages ? So do they sign these ledger heights ? If they do how often do they resign. Every time they broadcast ?
yacovm (Mon, 02 Apr 2018 16:21:05 GMT):
yeah they sign alive messages
yacovm (Mon, 02 Apr 2018 16:21:11 GMT):
how often - its configurable
yacovm (Mon, 02 Apr 2018 16:21:20 GMT):
are you worried that HSM is costly or something?
yacovm (Mon, 02 Apr 2018 16:21:45 GMT):
it should be one in a few seconds
rsherwood (Mon, 02 Apr 2018 16:22:18 GMT):
HSMs only support a certain number of signing operations per second depending on the key types and lengths used.
yacovm (Mon, 02 Apr 2018 16:22:31 GMT):
well but we sign once every few seconds
yacovm (Mon, 02 Apr 2018 16:22:38 GMT):
so it's a fraction per second
yacovm (Mon, 02 Apr 2018 16:22:50 GMT):
like, for example- once in 5 seconds
rsherwood (Mon, 02 Apr 2018 16:22:51 GMT):
Is that once per second per channel ?
yacovm (Mon, 02 Apr 2018 16:22:54 GMT):
i think that's the default
yacovm (Mon, 02 Apr 2018 16:23:04 GMT):
so we have alive messages that are global and cross channel
yacovm (Mon, 02 Apr 2018 16:23:09 GMT):
and special messages per channel
yacovm (Mon, 02 Apr 2018 16:23:21 GMT):
think of them as alive messages per channel
rsherwood (Mon, 02 Apr 2018 16:24:03 GMT):
So if we have 13 channels that's 13 signings per "configured interval" per peer.
rsherwood (Mon, 02 Apr 2018 16:24:57 GMT):
Are there keepalives on the peer to order coms as well ?
yacovm (Mon, 02 Apr 2018 16:26:09 GMT):
well
yacovm (Mon, 02 Apr 2018 16:26:12 GMT):
if you have 13 channels
yacovm (Mon, 02 Apr 2018 16:26:19 GMT):
just increase the time of the state info message sending
yacovm (Mon, 02 Apr 2018 16:26:26 GMT):
to like - once in 13 seconds
yacovm (Mon, 02 Apr 2018 16:26:39 GMT):
but you need to do it globally across channel ideally
yacovm (Mon, 02 Apr 2018 16:26:46 GMT):
to all peers in the channel
yacovm (Mon, 02 Apr 2018 16:27:00 GMT):
so, between peers and orderers - there is a gRPC level keepalive
yacovm (Mon, 02 Apr 2018 16:27:04 GMT):
so no ECDSA signing
yacovm (Mon, 02 Apr 2018 16:27:13 GMT):
it's like a tcp level ping-pong
yacovm (Mon, 02 Apr 2018 16:27:25 GMT):
and there is a connection for each channel between peer and orderer
rsherwood (Mon, 02 Apr 2018 16:29:37 GMT):
So we just get a single setup signed message per channel per peer between peer and orderer and after that messages (but encrypted at the TCP level).
yacovm (Mon, 02 Apr 2018 16:30:54 GMT):
yeah exactly!
yacovm (Mon, 02 Apr 2018 16:30:58 GMT):
to be more specific:
yacovm (Mon, 02 Apr 2018 16:31:15 GMT):
the peer sends to the orderer- "hey, can you start giving me block for channel X starting from sequence N?"
yacovm (Mon, 02 Apr 2018 16:31:20 GMT):
the orderer checks the certificate
yacovm (Mon, 02 Apr 2018 16:31:28 GMT):
either complies or rejects
yacovm (Mon, 02 Apr 2018 16:31:40 GMT):
and starting from that point - the connection is encrypted via TLS
yacovm (Mon, 02 Apr 2018 16:31:46 GMT):
*per channel*
yacovm (Mon, 02 Apr 2018 16:31:58 GMT):
if you're concerned about HSM performance
yacovm (Mon, 02 Apr 2018 16:32:06 GMT):
what you *really* need to be afraid of
yacovm (Mon, 02 Apr 2018 16:32:14 GMT):
is the validation of transactions :(
yacovm (Mon, 02 Apr 2018 16:32:19 GMT):
imagine a block that has 150 transactions
yacovm (Mon, 02 Apr 2018 16:32:51 GMT):
the peer, validates them `n` at a time, when `n` is number of cores
yacovm (Mon, 02 Apr 2018 16:33:08 GMT):
but, if you can only do `x` hsm operations in a second
yacovm (Mon, 02 Apr 2018 16:33:17 GMT):
then.... you know
yacovm (Mon, 02 Apr 2018 16:33:27 GMT):
depends on how big `x` is
yacovm (Mon, 02 Apr 2018 16:33:35 GMT):
what is its estimated size btw?
yacovm (Mon, 02 Apr 2018 16:33:48 GMT):
how many HSM signature verification can you do in a second?
rsherwood (Mon, 02 Apr 2018 16:40:04 GMT):
We have 4 gemalto HSMs (for availability) . So we can do a lot. I forget the numbers. When we calculated we allowed for a signature for the submitter & 3 endorsers per transactions and 3 orderers per block. But did not know about the gossip signatures.
yacovm (Mon, 02 Apr 2018 16:41:33 GMT):
and i'm telling you that gossip signatures should be the least of your worries
yacovm (Mon, 02 Apr 2018 16:41:45 GMT):
you should worry about validation of transactions
yacovm (Mon, 02 Apr 2018 16:41:49 GMT):
that's where the big money is
rsherwood (Mon, 02 Apr 2018 16:43:04 GMT):
Given that we don't have anchor peers specified , and are on 1.0.6 and only have one peer per organisation , how do I make sure that the peer does not attempt to sign gossip blocks unnecessarily ?
rsherwood (Mon, 02 Apr 2018 16:43:48 GMT):
Do you know if there is a variable that gets checks to say "don't bother with gossip" ?
yacovm (Mon, 02 Apr 2018 16:44:02 GMT):
if you don't have anchor peers and don't have bootstrap peers too
yacovm (Mon, 02 Apr 2018 16:44:05 GMT):
then there is no gossip
yacovm (Mon, 02 Apr 2018 16:44:16 GMT):
and peers don't sign blocks
yacovm (Mon, 02 Apr 2018 16:44:19 GMT):
they sign messages
rsherwood (Mon, 02 Apr 2018 16:46:44 GMT):
Does the fabric work that out before signing the height of its channel or only afterwards ?
rsherwood (Mon, 02 Apr 2018 16:46:44 GMT):
Does the fabric work that out before signing the height of its channel or only afterwards ? We have had a problem with the HSMs and CPU and in the stack trace I did see calls relating to example and I was juts a bit worried we might still be signing. gossip. goroutine 182 [select]:
github.com/hyperledger/fabric/gossip/gossip/channel.(*gossipChannel).periodicalInvocation(0xc4216c70e0, 0xc42145f4d0, 0xc4202f0960)
/opt/gopath/src/github.com/hyperledger/fabric/gossip/gossip/channel/channel.go:223 +0x12c
created by github.com/hyperledger/fabric/gossip/gossip/channel.NewGossipChannel
/opt/gopath/src/github.com/hyperledger/fabric/gossip/gossip/channel/channel.go:204 +0x81e
grapebaba (Tue, 03 Apr 2018 03:57:43 GMT):
hi, anyone here? seems i got a gossip tls issue
grapebaba (Tue, 03 Apr 2018 03:58:11 GMT):
Clipboard - 2018年4月3日中午12点03分
grapebaba (Tue, 03 Apr 2018 03:58:20 GMT):
Clipboard - 2018年4月3日中午12点03分
grapebaba (Tue, 03 Apr 2018 03:58:32 GMT):
Clipboard - 2018年4月3日中午12点03分
yacovm (Tue, 03 Apr 2018 07:15:57 GMT):
Well that's a DNS SAN not an ip SAN
yacovm (Tue, 03 Apr 2018 07:16:01 GMT):
@grapebaba
grapebaba (Tue, 03 Apr 2018 07:17:41 GMT):
how to fix this issue?
grapebaba (Tue, 03 Apr 2018 07:17:58 GMT):
is it a wrong configuration
yacovm (Tue, 03 Apr 2018 08:03:25 GMT):
hmmm, do your anchor peers have IP addresses?
grapebaba (Tue, 03 Apr 2018 11:06:54 GMT):
all use domain
yacovm (Tue, 03 Apr 2018 11:07:03 GMT):
:(
Rumeel_Hussain (Tue, 03 Apr 2018 15:04:09 GMT):
Has joined the channel.
grapebaba (Wed, 04 Apr 2018 07:36:43 GMT):
@yacovm in my environment two org both have three peers, if i set CORE_PEER_GOSSIP_BOOTSTRAP=peer0 for all three peers each org, the error will raise
grapebaba (Wed, 04 Apr 2018 07:37:44 GMT):
if i remove CORE_PEER_GOSSIP_BOOTSTRAP=peer0 and have all peers set CORE_PEER_GOSSIP_EXTERNALENDPOINT, the error will disappear
grapebaba (Wed, 04 Apr 2018 07:38:05 GMT):
but i only want to set CORE_PEER_GOSSIP_EXTERNALENDPOINT for anchor peer
yacovm (Wed, 04 Apr 2018 08:33:37 GMT):
you're setting `peer0` for both orgs?!! @grapebaba
grapebaba (Wed, 04 Apr 2018 08:33:45 GMT):
no
grapebaba (Wed, 04 Apr 2018 08:34:12 GMT):
each org have a peer0
yacovm (Wed, 04 Apr 2018 08:35:01 GMT):
oh ok
yacovm (Wed, 04 Apr 2018 08:35:11 GMT):
but it needs to be `peer0.org2tac-com` etc.
yacovm (Wed, 04 Apr 2018 08:35:14 GMT):
not just "peer0"
grapebaba (Wed, 04 Apr 2018 08:35:20 GMT):
yeah
grapebaba (Wed, 04 Apr 2018 08:35:37 GMT):
exactly
yacovm (Wed, 04 Apr 2018 08:37:53 GMT):
do you have `peer.gossip.endpoint` configured?
Ammu (Wed, 04 Apr 2018 13:47:42 GMT):
i am just doing BYFN, Create & "Join Channel " up to this step i have done, so when i shut down my system, from staring i need to do?? following this link http://hyperledger-fabric.readthedocs.io/en/latest/build_network.html#create-join-channel
richzhao (Wed, 04 Apr 2018 15:56:25 GMT):
Has joined the channel.
grapebaba (Thu, 05 Apr 2018 15:08:56 GMT):
@yacovm no, what's that?
yacovm (Thu, 05 Apr 2018 15:09:22 GMT):
hmmm
yacovm (Thu, 05 Apr 2018 15:09:32 GMT):
there has to be some ip address configured instead of a host...
yacovm (Thu, 05 Apr 2018 15:09:37 GMT):
somewhere
Ryan2 (Sun, 08 Apr 2018 07:15:43 GMT):
Hi , I got a confusion on CORE_PEER_GOSSIP_BOOTSTRAP
example, The Org0 has 2 peer peer0 and peer1
why starting peer0 have to bootstrap gossip peer1 and vice verse as below
peer0.org1.example.com:
environment:
- CORE_PEER_GOSSIP_BOOTSTRAP=peer1.org1.example.com:7051
peer1.org1.example.com:
environment:
- CORE_PEER_GOSSIP_BOOTSTRAP=peer0.org1.example.com:7051
https://github.com/hyperledger/fabric-test/blob/master/feature/docker-compose/docker-compose-kafka.yml
Ryan2 (Sun, 08 Apr 2018 07:15:43 GMT):
Hi , I got a confusion on CORE_PEER_GOSSIP_BOOTSTRAP
example, The Org0 has 2 peer peer0 and peer1
why starting peer0 have to bootstrap gossip peer1 and vice verse as below
"peer0.org1.example.com:
environment:
- CORE_PEER_GOSSIP_BOOTSTRAP=peer1.org1.example.com:7051
peer1.org1.example.com:
environment:
- CORE_PEER_GOSSIP_BOOTSTRAP=peer0.org1.example.com:7051"
(source: https://github.com/hyperledger/fabric-test/blob/master/feature/docker-compose/docker-compose-kafka.yml)
Ryan2 (Sun, 08 Apr 2018 14:44:21 GMT):
Hi, I got a issue, when adding new Org into network,
after running join command, new peer (belong to newly added Org) got error
`[31m2018-04-08 14:23:31.860 UTC [gossip/comm] authenticateRemotePeer -> ERRO 056[0m Failed verifying signature from 127.32.1.131:7051 : Failed to reach implicit threshold of 1 sub-policies, required 1 remaining`
(127.32.1.131 is peer 0 on Org0)
How do fix the error?
Ryan2 (Sun, 08 Apr 2018 14:44:21 GMT):
Hi, I got an issue, when adding new Org into network,
after running join command, new peer (belong to newly added Org) got error
`[31m2018-04-08 14:23:31.860 UTC [gossip/comm] authenticateRemotePeer -> ERRO 056[0m Failed verifying signature from 127.32.1.131:7051 : Failed to reach implicit threshold of 1 sub-policies, required 1 remaining`
(127.32.1.131 is peer 0 on Org0)
How do fix the error?
Ryan2 (Sun, 08 Apr 2018 14:44:21 GMT):
Hi, I got an issue while adding new Org into network,
after running join command, new peer (belong to newly added Org) got error
`[31m2018-04-08 14:23:31.860 UTC [gossip/comm] authenticateRemotePeer -> ERRO 056[0m Failed verifying signature from 127.32.1.131:7051 : Failed to reach implicit threshold of 1 sub-policies, required 1 remaining`
(127.32.1.131 is peer 0 on Org0)
How do fix the error?
Ryan2 (Sun, 08 Apr 2018 14:44:21 GMT):
Hi,
Q1: I got an issue while adding new Org into network,
after running join command, new peer (belong to newly added Org) got error
`[31m2018-04-08 14:23:31.860 UTC [gossip/comm] authenticateRemotePeer -> ERRO 056[0m Failed verifying signature from 127.32.1.131:7051 : Failed to reach implicit threshold of 1 sub-policies, required 1 remaining`
(127.32.1.131 is peer 0 on Org0)
How do fix the error?
Q2: Hi, My Org0 have peer0 and peer1, peer0 start bootstrap peer1 and peer1 start bootstrap peer0, but I got error msg as below, Why this happens and how to fix?
2018-04-09 04:54:29.447 UTC [gossip/comm] authenticateRemotePeer -> WARN 04b Identity store rejected 175.32.1.147:40470 : Peer Identity [0a 07 4f 72 67 31 4d 53 ... 50 12 c7 06 2d 2d 2d 2d ] cannot be validated. No MSP found able to do that.
2018-04-09 04:54:29.447 UTC [gossip/comm] GossipStream -> ERRO 04c Authentication failed: Peer Identity [0a 07 4f 72 67 31 4d 53 50 12 ..4e 20 43 45 52 54 49 46 43 50 5] cannot be validated. No MSP found able to do that.
help me out!
Ryan2 (Sun, 08 Apr 2018 14:44:21 GMT):
Hi,
Q1: I got an issue while adding new Org into network,
after running join command, new peer (belong to newly added Org) got error
`[31m2018-04-08 14:23:31.860 UTC [gossip/comm] authenticateRemotePeer -> ERRO 056[0m Failed verifying signature from 127.32.1.131:7051 : Failed to reach implicit threshold of 1 sub-policies, required 1 remaining`
(127.32.1.131 is peer 0 on Org0)
How do fix the error?
Q2: Hi, My Org0 have peer0 and peer1, peer0 start bootstrap peer1 and peer1 start bootstrap peer0, but I got error msg as below, Why this happens and how to fix?
2018-04-09 04:54:29.447 UTC [gossip/comm] authenticateRemotePeer -> WARN 04b Identity store rejected 175.32.1.147:40470 : Peer Identity [0a 07 4f 72 67 31 4d 53 ... 50 12 c7 06 2d 2d 2d 2d ] cannot be validated. No MSP found able to do that.
2018-04-09 04:54:29.447 UTC [gossip/comm] GossipStream -> ERRO 04c Authentication failed: Peer Identity [0a 07 4f 72 67 31 4d 53 50 12 ..4e 20 43 45 52 54 49 46 43 50 5] cannot be validated. No MSP found able to do that.
Ryan2 (Sun, 08 Apr 2018 14:44:21 GMT):
Hi,
I got an issue while adding new Org into network,
after running join command, new peer (belong to newly added Org) got error
`[31m2018-04-08 14:23:31.860 UTC [gossip/comm] authenticateRemotePeer -> ERRO 056[0m Failed verifying signature from 127.32.1.131:7051 : Failed to reach implicit threshold of 1 sub-policies, required 1 remaining`
(127.32.1.131 is peer 0 on Org0)
How do fix the error?
davidgsmits (Thu, 12 Apr 2018 13:49:11 GMT):
Has joined the channel.
neharprodduturi (Mon, 16 Apr 2018 06:54:03 GMT):
Has joined the channel.
AnilOner (Mon, 16 Apr 2018 13:36:53 GMT):
Has joined the channel.
Ammu (Tue, 17 Apr 2018 10:14:55 GMT):
Capture.PNG
tushad (Tue, 17 Apr 2018 17:32:57 GMT):
Has joined the channel.
NoorFairoza (Tue, 17 Apr 2018 18:08:25 GMT):
Has joined the channel.
bh4rtp (Wed, 18 Apr 2018 00:54:37 GMT):
it is assumed that they can decode the messages online without public keys.
dsl (Thu, 19 Apr 2018 04:24:54 GMT):
Has joined the channel.
dsl (Thu, 19 Apr 2018 07:42:55 GMT):
Hi,
I am relatively new to fabric. I have gone through the documentation and I am working with a few samples in hand. I am not able to find out a configuration file which is mentioning the gossip protocol for the network.
1) Can anyone please guide me to find out where I can do the configuration?
2) Also where can I find the list of peers in the organisation to whom the leader can communicate?
yacovm (Thu, 19 Apr 2018 08:13:47 GMT):
1) https://github.com/hyperledger/fabric/blob/release-1.1/sampleconfig/core.yaml#L130
yacovm (Thu, 19 Apr 2018 08:14:09 GMT):
2) peers in the org communicate with all peers in the same org
dsl (Thu, 19 Apr 2018 12:00:20 GMT):
@yacovm Thanks. But none of the samples seems to have such a file in it.
joaquimpedrooliveira (Thu, 19 Apr 2018 13:40:32 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=s6hNGFgS5btF46LeW) @dsl answered to you in private. What file is this that you are looking for?
asaningmaxchain123 (Thu, 19 Apr 2018 16:07:08 GMT):
Has joined the channel.
asaningmaxchain123 (Thu, 19 Apr 2018 16:22:50 GMT):
@yacovm the peer use the gossip to sync ledger data,the range is org-scoped or channel-scoped?
yacovm (Thu, 19 Apr 2018 16:58:42 GMT):
@asaningmaxchain123 channel
asaningmaxchain123 (Thu, 19 Apr 2018 23:17:52 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=eZvq9EQFiWtvb2LMX) @yacovm but here?
yacovm (Fri, 20 Apr 2018 10:28:36 GMT):
they communicate
yacovm (Fri, 20 Apr 2018 10:28:46 GMT):
but that doesn't mean they send blocks
asaningmaxchain123 (Sun, 22 Apr 2018 04:57:28 GMT):
what's the propose of the `peers in the org communicate with all peers in the same org`
hrt031293 (Mon, 23 Apr 2018 11:37:45 GMT):
Has joined the channel.
Ryan2 (Mon, 23 Apr 2018 23:45:49 GMT):
Hi, can you know how to avoid this
https://chat.hyperledger.org/channel/fabric-orderer?msg=ZJwjPBQtZeRQExNdn
Ryan2 (Mon, 23 Apr 2018 23:45:49 GMT):
Hi, do you know how to avoid this
https://chat.hyperledger.org/channel/fabric-orderer?msg=ZJwjPBQtZeRQExNdn
yacovm (Tue, 24 Apr 2018 00:05:16 GMT):
answered you in #fabric-questions @Ryan2
Ryan2 (Tue, 24 Apr 2018 00:05:44 GMT):
thank you @yacovm
hosemose (Tue, 24 Apr 2018 04:59:05 GMT):
Has joined the channel.
MeenakshiSingh (Fri, 27 Apr 2018 11:02:23 GMT):
hi..can we define multiple peers for bootstrap within the same organization against the `CORE_PEER_GOSSIP_BOOTSTRAP`. i.e., if there are n peers in an organization then for each peer mention n-1 other peers as bootstrap peers irrespective of which one is the anchor peer or the leader peer for the organization?
yacovm (Fri, 27 Apr 2018 12:13:13 GMT):
@MeenakshiSingh yeah - just separate them with spaces
yacovm (Fri, 27 Apr 2018 12:13:15 GMT):
it should work
yacovm (Fri, 27 Apr 2018 12:13:37 GMT):
this code actually treats this as an array and not as a single string
MeenakshiSingh (Fri, 27 Apr 2018 12:40:54 GMT):
ok.. Also, how does the gossip ensure byzantine failures as mentioned here: ```Gossip messaging is continuous, and
each peer on a channel is constantly receiving current and consistent ledger data from multiple peers. Each gossiped
message is signed, thereby allowing Byzantine participants sending faked messages to be easily identified and the
distribution of the message(s) to unwanted targets to be prevented. Peers affected by delays, network partitions, or
other causes resulting in missed blocks will eventually be synced up to the current ledger state by contacting peers in
possession of these missing blocks```
yacovm (Fri, 27 Apr 2018 12:50:39 GMT):
it is mentioned how
MeenakshiSingh (Fri, 27 Apr 2018 13:11:42 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=nWjCjQxLYdShDxGti) @yacovm Just separating with spaces isn't working. While sourcing CORE_PEER_GOSSIP_BOOTSTRAP=ip1:7051 ip2:7051 I am getting an error at ip2 sying invalid identifier.
yacovm (Fri, 27 Apr 2018 13:14:21 GMT):
it works for me....
yacovm (Fri, 27 Apr 2018 13:14:24 GMT):
```
2018-04-27 16:13:59.572 IDT [gossip/discovery] func1 -> WARN 022 Could not connect to {p2:7051 [] [] p2:7051
yacovm (Fri, 27 Apr 2018 13:14:53 GMT):
Clipboard - April 27, 2018 4:14 PM
MeenakshiSingh (Fri, 27 Apr 2018 13:16:41 GMT):
can we not do this in environment variables?
yacovm (Fri, 27 Apr 2018 13:18:00 GMT):
you need `"`
yacovm (Fri, 27 Apr 2018 13:18:15 GMT):
`"p1 p2"`
MeenakshiSingh (Fri, 27 Apr 2018 13:20:17 GMT):
thanks...that worked
chainsaw (Fri, 27 Apr 2018 15:51:45 GMT):
Has joined the channel.
MeenakshiSingh (Fri, 27 Apr 2018 17:51:33 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=337NBH37ggAKk58hR) As for my earlier question, could you please elaborate on a few points:
1. `each peer on a channel is constantly receiving current and consistent ledger data from multiple peers.` --> does the gossip message encapsulate all the data present in the ledger or only the world state..that is to say that gossip messages ensure ledger consistency/validation or transaction/worldstate validation? If its entire ledger then wouldn't gossip messages themselves become too heavy and add to latency?
2. How does the Gossip messages ensure sync between the peers? Can two lagging peers who haapen to have same number of blocks, will be allowed to participate in endorsement?
3. Now, since the gossip messages are signed, we can trust the handshake and hence any byzantine participants can be identified. However, how is the trust established as the peers do not possess certificates of other peers?
4. If in a network of 4 nodes (2 peers per org), if certificate of two peers, one in each org is compromised and the EP is AND(Org1MSP.member, Org2MSP.member), would gossip be able to identify this?
5. Is there a rule of thumb on how to securely define EPs based on the network?
MeenakshiSingh (Fri, 27 Apr 2018 17:51:33 GMT):
@yacovm [ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=337NBH37ggAKk58hR) As for my earlier question, could you please elaborate on a few points:
1. `each peer on a channel is constantly receiving current and consistent ledger data from multiple peers.` --> does the gossip message encapsulate all the data present in the ledger or only the world state..that is to say that gossip messages ensure ledger consistency/validation or transaction/worldstate validation? If its entire ledger then wouldn't gossip messages themselves become too heavy and add to latency?
2. How does the Gossip messages ensure sync between the peers? Can two lagging peers who haapen to have same number of blocks, will be allowed to participate in endorsement?
3. Now, since the gossip messages are signed, we can trust the handshake and hence any byzantine participants can be identified. However, how is the trust established as the peers do not possess certificates of other peers?
4. If in a network of 4 nodes (2 peers per org), if certificate of two peers, one in each org is compromised and the EP is AND(Org1MSP.member, Org2MSP.member), would gossip be able to identify this?
5. Is there a rule of thumb on how to securely define EPs based on the network?
yacovm (Fri, 27 Apr 2018 17:53:54 GMT):
we replicate only ledger
yacovm (Fri, 27 Apr 2018 17:54:29 GMT):
> If its entire ledger then wouldn't gossip messages themselves become too heavy and add to latency
well, no one said it's cheap.
MeenakshiSingh (Fri, 27 Apr 2018 18:13:50 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=RJn7mWkZ7rHv4wbxJ) @yacovm ok, so this is for the lagging peer to get the missing blocks. This is one aspect of gossip, I presume. How do they really ensure sync, does this also require bundling of the entire data?
yacovm (Fri, 27 Apr 2018 18:14:39 GMT):
block by block
yacovm (Fri, 27 Apr 2018 18:14:45 GMT):
in batches
MeenakshiSingh (Fri, 27 Apr 2018 18:16:39 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=Dh87c4jiQnQfX5oEf) @yacovm ok.
MeenakshiSingh (Fri, 27 Apr 2018 18:18:33 GMT):
Could you please explain these as well. I would really be helpful.
MeenakshiSingh (Fri, 27 Apr 2018 18:18:33 GMT):
Could you please explain these as well. It would really be helpful.
MeenakshiSingh (Fri, 27 Apr 2018 18:19:21 GMT):
*2*. Can two lagging peers who haapen to have same number of blocks, will be allowed to participate in endorsement?
*3*. Now, since the gossip messages are signed, we can trust the handshake and hence any byzantine participants can be identified. However, how is the trust established as the peers do not possess certificates of other peers?
*4*. If in a network of 4 nodes (2 peers per org), if certificate of two peers, one in each org is compromised and the EP is AND(Org1MSP.member, Org2MSP.member), would gossip be able to identify this?
*5*. Is there a rule of thumb on how to securely define EPs based on the network?
yacovm (Fri, 27 Apr 2018 18:20:01 GMT):
3) the rule is simple... 2 peers can talk to each other, only if they are either in the same organization (have same root CA) or they are both in some same channel
MeenakshiSingh (Mon, 30 Apr 2018 09:12:24 GMT):
Hi...I wanted to test a couple of samples with side db feature. Can somebody point me to its documentation or a getting started guide.
C0rWin (Mon, 30 Apr 2018 13:43:12 GMT):
@MeenakshiSingh please ask on #fabric-ledger channel
JackStrohm (Mon, 30 Apr 2018 16:05:12 GMT):
Has joined the channel.
JackStrohm (Mon, 30 Apr 2018 17:04:12 GMT):
I’ve got 3 peers but they are having trouble gossiping. Can anyone help me?
peer0 gives me this:
peer0.org1.example.com | 2018-04-30 16:47:26.794 UTC [gossip/service] func1 -> INFO 06c Initialize gossip with endpoint peer0.org1.example.com:7051 and bootstrap set [peer0.org1.example.com:7051]
peer0.org1.example.com | 2018-04-30 16:47:26.796 UTC [gossip/discovery] NewDiscoveryService -> INFO 076 Started { [] [115 248 30 86 68 248 12 185 3 149 23 119 90 20 155 83 143 82 117 213 199 184 228 141 219 75 110 189 246 183 219 86] peer0.org1.example.com:7051
acbellini (Tue, 01 May 2018 21:10:25 GMT):
Has joined the channel.
kevin-s-wang (Thu, 03 May 2018 02:38:10 GMT):
Has joined the channel.
hrt031293 (Mon, 07 May 2018 11:38:07 GMT):
Hello everyone,
Did anyone present her, done the `Adding an Org to a Channel`??
I m facing a problem...
lclclc (Tue, 08 May 2018 12:52:44 GMT):
Has joined the channel.
migrenaa (Tue, 08 May 2018 13:18:14 GMT):
Has joined the channel.
gut (Tue, 08 May 2018 14:43:25 GMT):
Has joined the channel.
gut (Tue, 08 May 2018 14:48:43 GMT):
@yacovm "2 peers can talk to each other, only if they are either in the same organization (have same root CA)"
Does it mean that if two orgs have the same rootCA (e.g. Let's Encrypt in a future) their peers will be supposed to be in the same org?
yacovm (Tue, 08 May 2018 14:58:40 GMT):
same root CA... and as long as you don't have OUs
lclclc (Wed, 09 May 2018 02:36:25 GMT):
Does the orderer keeps all history data in its ledger(I read some documents and found this module, haven't check in repo)? Can a leader peer pull all blocks only from orderer to catch up latest world state, or does it need to talk to other organizations' leader peer? Any sources or document would be helpful.
gut (Wed, 09 May 2018 11:40:30 GMT):
[Actually, I have same root and each org has its own OU, but individuals are not configured yet :sweat_smile:](https://chat.hyperledger.org/channel/fabric-gossip?msg=pAeD8o79sHKX2ZuH7) @yacovm
hrt031293 (Wed, 09 May 2018 11:57:33 GMT):
Hello everyone,
I want to know that, if I want to create a project in fabric, then what should be the basic architecture?
Thanks for the help
akshaylawange001 (Wed, 09 May 2018 19:38:03 GMT):
Has joined the channel.
gut (Thu, 10 May 2018 08:41:29 GMT):
@hrt031293 If you mean a very basic one to start, you can check "byfn" configuration. Nevertheless, maybe #fabric-questions is the channel where they can help you better.
hrt031293 (Thu, 10 May 2018 10:39:58 GMT):
@gut OK,thanks
hrt031293 (Thu, 10 May 2018 10:40:47 GMT):
Is there anyone, who can explain me that what's going on, in this topic
`http://hyperledger-fabric.readthedocs.io/en/latest/channel_update_tutorial.html#add-the-org3-crypto-material`??
hrt031293 (Thu, 10 May 2018 13:12:42 GMT):
Screenshot from 2018-05-10 18-03-05.png
hrt031293 (Thu, 10 May 2018 13:12:42 GMT):
Screenshot from 2018-05-10 18-03-05.png
divyank (Thu, 10 May 2018 15:26:26 GMT):
Was wondering if anyone here can help with https://jira.hyperledger.org/browse/FAB-10000
divyank (Thu, 10 May 2018 15:26:26 GMT):
Was wondering if anyone here can help with FAB-10000
yacovm (Thu, 10 May 2018 15:28:29 GMT):
@divyank - I can't figure out the environment from the text in `Environment`
yacovm (Thu, 10 May 2018 15:28:39 GMT):
can you just pour everything into the JIRA description?
divyank (Thu, 10 May 2018 15:29:26 GMT):
Sure
troyronda (Thu, 10 May 2018 15:43:20 GMT):
way to hit issue # 10,000 :)
troyronda (Thu, 10 May 2018 15:43:20 GMT):
way to hit jira # 10,000 :)
yacovm (Thu, 10 May 2018 15:48:05 GMT):
@troyronda yeah... I'm honored that JIRA 10,000 is gossip ;)
troyronda (Thu, 10 May 2018 15:48:17 GMT):
lol
yacovm (Thu, 10 May 2018 15:48:51 GMT):
BTW - I had a chat with @divyank - once he posts the docker-compose he's using to reproduce to JIRA i'll immediately take a look (and solve it)
yacovm (Thu, 10 May 2018 15:49:05 GMT):
also i implemented collection support for service discovery https://gerrit.hyperledger.org/r/#/c/21645/
troyronda (Thu, 10 May 2018 15:49:05 GMT):
cool, thanks
yacovm (Thu, 10 May 2018 15:49:12 GMT):
all left now is cc2cc support :)
yacovm (Thu, 10 May 2018 15:49:27 GMT):
but i still need to update the client implementation in go
yacovm (Thu, 10 May 2018 15:49:37 GMT):
to expose an API for that...
troyronda (Thu, 10 May 2018 15:50:42 GMT):
sure, looking forward to full support in the Go SDK.
troyronda (Thu, 10 May 2018 15:50:42 GMT):
yup, looking forward to full support in the Go SDK.
akshaylawange001 (Thu, 10 May 2018 18:04:50 GMT):
Hi, I am getting error when peers try to get in sync after starting the network. The network starts perfectly, but when we look at one of the org log we got this :
akshaylawange001 (Thu, 10 May 2018 18:04:53 GMT):
`2018-05-10 17:41:47.714 UTC [gossip/comm] func1 -> WARN 2398 peer1.org1.example.com:7051, PKIid:[78 228 20 118 241 29 246 96 60 245 91 194 173 188 52 134 131 124 240 240 36 224 151 8 63 73 242 97 100 239 233 188] isn't responsive: EOF
2018-05-10 17:41:47.715 UTC [gossip/discovery] expireDeadMembers -> WARN 2399 Entering [[78 228 20 118 241 29 246 96 60 245 91 194 173 188 52 134 131 124 240 240 36 224 151 8 63 73 242 97 100 239 233 188]]
2018-05-10 17:41:47.715 UTC [gossip/discovery] expireDeadMembers -> WARN 239a Closing connection to Endpoint: peer1.org1.example.com:7051, InternalEndpoint: , PKI-ID: [78 228 20 118 241 29 246 96 60 245 91 194 173 188 52 134 131 124 240 240 36 224 151 8 63 73 242 97 100 239 233 188], Metadata: []
2018-05-10 17:41:47.715 UTC [gossip/discovery] expireDeadMembers -> WARN 239b Exiting
2018-05-10 17:41:49.205 UTC [gossip/gossip] Gossip -> WARN 239c Failed obtaining gossipChannel of [114 101 112 111 51 99 104 97 110 110 101 108] aborting
`
akshaylawange001 (Thu, 10 May 2018 18:05:51 GMT):
can anyone explain?
hrt031293 (Mon, 14 May 2018 06:34:49 GMT):
Hi everyone,
Anyone please tell me, that is it necessary to convert the json files into protobuf (.pb) files, in the "Adding an Org to a Channel" tutorial??
hrt031293 (Mon, 14 May 2018 08:06:40 GMT):
Hi everyone,
Anyone please tell me, that is it necessary to convert the json files into protobuf (.pb) files, in the "Adding an Org to a Channel" tutorial??
versus (Mon, 14 May 2018 09:05:48 GMT):
Has joined the channel.
IgorSim (Mon, 14 May 2018 09:05:53 GMT):
Has joined the channel.
buridiaditya (Tue, 15 May 2018 08:21:25 GMT):
Has joined the channel.
gut (Tue, 15 May 2018 08:27:41 GMT):
Hi. I've got a theoretical error. Is not directly related to the gossip service, but with race conditions on endorsement. Let's say...
If there are two endorsers belonging to different orgs that are acting over the same variable, that variable is modified over the same state. I think, one of them is going to have an error on commitment and the winner will successfully modify the variable state. But:
1. Which error is going to have the first client?
2. From who?
3. Is there any standardized mechanism to assure the order of commitment?
sauravverma (Tue, 15 May 2018 10:30:08 GMT):
Has joined the channel.
asaningmaxchain123 (Tue, 15 May 2018 16:10:25 GMT):
@yacovm @C0rWin if the `externalEndpoint` doesn't set,it doesn't use gossip to spread msg cross the orgs?
yacovm (Tue, 15 May 2018 16:12:57 GMT):
if you don't set an external endpoint to a peer then it is only visible to its own organization
asaningmaxchain123 (Tue, 15 May 2018 16:22:55 GMT):
Clipboard - May 16, 2018 12:22 AM
asaningmaxchain123 (Tue, 15 May 2018 16:24:22 GMT):
in the e2e_cli example , it use the the *anchors.tx to set the `externalEndpointe`
asaningmaxchain123 (Tue, 15 May 2018 16:24:32 GMT):
?
asaningmaxchain123 (Tue, 15 May 2018 16:32:13 GMT):
the peer will use the *anchor.tx to update the `AnchorPeer`
asaningmaxchain123 (Tue, 15 May 2018 16:32:13 GMT):
from source code,the peer will use the *anchor.tx to update the `AnchorPeer`
asaningmaxchain123 (Tue, 15 May 2018 16:32:27 GMT):
Clipboard - May 16, 2018 12:32 AM
asaningmaxchain123 (Wed, 16 May 2018 00:16:11 GMT):
if one channel create by two org,the orgs can't communicate with each other?
asaningmaxchain123 (Wed, 16 May 2018 14:45:18 GMT):
@yacovm @C0rWin i want to know the fabric how to use the gossip? so i have some question, 1) how to know each other,2)how to got the block from other peer
C0rWin (Wed, 16 May 2018 23:51:32 GMT):
@asaningmaxchain123 have you seen this one? http://hyperledger-fabric.readthedocs.io/en/release-1.1/gossip.html
asaningmaxchain123 (Wed, 16 May 2018 23:51:52 GMT):
yes
gut (Fri, 18 May 2018 10:38:34 GMT):
Has left the channel.
JulesMiller (Mon, 21 May 2018 01:42:17 GMT):
Has joined the channel.
lclclc (Mon, 21 May 2018 12:30:11 GMT):
Do anchor peers have to know and communicate with each other directly? I have a consortium network distributed in 2 machines. Machine 1 has the kafka orderers and org1 peers, machine2 has org2 peers. Both machine has different docker-compose bootstrapped networks and they don't use Swarm or something to communicate with other. Machine2's docker-compose file has `extra-hosts` to let org2 cli understand the public address of orderer containers, but I don't list public address of peer containers of org1. Then network is working fine, when I invoke chaincode in org1 cli, I can see the ledger is updated in org2. But the org2 peer's log has a lot of warning:
```
2018-05-21 11:48:56.328 UTC [gossip/discovery] func1 -> WARN 13d Could not connect to {peer0.didi.com:7051 [] [] peer0.didi.com:7051
lclclc (Mon, 21 May 2018 12:31:11 GMT):
If my guess is correct, when the consortium is large(take, for examle, 10 orgs), I have to pointed out every org's anchor peer in every machine's docker-compose file?
buridiaditya (Mon, 21 May 2018 13:01:11 GMT):
I went through the code and found that the `InitGossipService` in gossip_service.go does not send request to pull missing blocks rather it accepts any incoming gossip messages and sends membership request. Can anybody tell me where to look at and what initiates the thread to send a request message to pull missing blocks from broadcast peers?
buridiaditya (Mon, 21 May 2018 13:01:11 GMT):
I went through the code and found that the `InitGossipService` in gossip_service.go does not send request to pull missing blocks rather it accepts any incoming gossip messages and sends membership request. Can anybody tell me where to look at and what initiates the thread to send a request message to pull missing blocks from leader/orderer/peer of same organisation?
buridiaditya (Tue, 22 May 2018 06:24:01 GMT):
Hey can anyone help me with this. I cannot figure out how the process of sharing missing blocks starts.
1. From Pull request - Cannot locate the code where such a request is created and sent.
2. Random Push of message blocks - Cannot locate this either.
When a peer starts it just sends the membership request and connects to bootstrap peers a thread for accepting blocks starts. Where is the code for checking if its ledger is lacking behind and sending a pull message requesting missing blocks ?? Please help.
asp 25 (Tue, 22 May 2018 19:29:25 GMT):
Has joined the channel.
Ammu (Wed, 23 May 2018 11:06:29 GMT):
medium.png
hrt031293 (Fri, 25 May 2018 11:39:35 GMT):
Can anyone tell me, that how can I use my local storage file as a chaincode, for the installation of the chaincode?
hrt031293 (Mon, 28 May 2018 09:28:52 GMT):
Hello,
What should be the value of the `-p` in the command, for installing the chaincode?
The command is "peer chaincode install".
sourishkrout (Tue, 29 May 2018 20:48:42 GMT):
Has joined the channel.
AlexanderZhovnuvaty (Wed, 30 May 2018 11:37:17 GMT):
Has joined the channel.
buridiaditya (Wed, 30 May 2018 12:39:08 GMT):
How is the state request gossip i.e by calling `RequestBlocksInRange` different from the pull-based gossip which uses pull engine
rogerwilcos (Wed, 30 May 2018 23:15:18 GMT):
Has joined the channel.
hrt031293 (Thu, 31 May 2018 12:12:09 GMT):
Screenshot from 2018-05-31 15-44-28.png
nagap (Thu, 31 May 2018 19:32:44 GMT):
Has joined the channel.
VinayChaudharyOfficial (Fri, 01 Jun 2018 04:55:23 GMT):
Has joined the channel.
Aswath8687 (Fri, 01 Jun 2018 20:09:24 GMT):
Has joined the channel.
demonkm (Sun, 03 Jun 2018 02:26:57 GMT):
Has joined the channel.
sambhavdutt (Tue, 05 Jun 2018 15:38:38 GMT):
Has joined the channel.
vish146 (Wed, 06 Jun 2018 18:16:53 GMT):
Has joined the channel.
abraham (Fri, 08 Jun 2018 05:43:29 GMT):
Has joined the channel.
paulananth (Fri, 15 Jun 2018 12:18:40 GMT):
Has joined the channel.
nvxtien (Fri, 22 Jun 2018 11:27:01 GMT):
Has joined the channel.
bh4rtp (Fri, 22 Jun 2018 12:34:54 GMT):
i am reviewing the peer log. there are numerous gossip messages. but from peer0.org1.example.com, i cannot find any gossip message from org2.example.com. how to explain it?
javrevasandeep (Mon, 25 Jun 2018 17:47:10 GMT):
@yacovm I am running a blockchain network which includes 2 orgs with 3 peers for each org in multihost environment. I am using fabric-ca to generate peers and orderer certificates. some days back all of the services got stopped suddenly. I restarted all of the services but unfortunately by mistake, i recreated the peer1-org1 instead of just restart it. So i fetch the 0 block first from peer1-org1 and then make the peer1-org1 to rejoin the existing channel, I installed the chaincode and upgraded the chaincode. Now I am getting the below warning messages in all of the peer logs
[gossip/comm] sendToEndpoint -> WARN 4892 Failed obtaining connection for peer1-org1:7051, PKIid:[130 169 200 147 151 98 112 245 155 234 176 88 178 195 88 136 118 154 94 91 90 37 108 90 11 182 83 100 124 97 165 32] reason: Authentication failure
2018-06-22 11:53:38.850 UTC [gossip/comm] createConnection -> WARN 4893 Remote endpoint claims to be a different peer, expected [130 169 200 147 151 98 112 245 155 234 176 88 178 195 88 136 118 154 94 91 90 37 108 90 11 182 83 100 124 97 165 32] but got [145 36 196 71 163 229 254 88 77 111 22 190 1 82 18 34 22 29 131 206 156 183 56 117 145 1 57 57 153 74 85 108]
javrevasandeep (Mon, 25 Jun 2018 17:50:06 GMT):
Do i need to update the anchor peers again since peer1-org1 rejoined the channel. And when i am updating the anchor peers, I am getting the below error message
Error: got unexpected status: BAD_REQUEST -- error authorizing update: error validating ReadSet: readset expected key [Group] /Channel/Application/org1 at version 0, but got version 1
javrevasandeep (Mon, 25 Jun 2018 17:53:34 GMT):
Below given is the channel config_block.json which i got with "peer channel fetch config $CONFIG_BLOCK_FILE -c $CHANNEL_NAME $ordererArgs"
https://hastebin.com/ruketageno.json
javrevasandeep (Mon, 25 Jun 2018 17:54:42 GMT):
could you please verify this channel config file and let me know whether you find any issue with this config file
yacovm (Mon, 25 Jun 2018 18:06:02 GMT):
Give the peers half an hour to adjust
yacovm (Mon, 25 Jun 2018 18:06:12 GMT):
@javrevasandeep
yacovm (Mon, 25 Jun 2018 18:06:33 GMT):
They need time to forget the old mapping
javrevasandeep (Mon, 25 Jun 2018 18:07:21 GMT):
yesterday i did the same. I waited for exactly half an hour and then restarted peer1-org1 but still could able to see these messages popping up again in each peer logs
javrevasandeep (Mon, 25 Jun 2018 18:07:21 GMT):
@yacovm yesterday i did the same. I waited for exactly half an hour and then restarted peer1-org1 but still could able to see these messages popping up again in each peer logs
javrevasandeep (Mon, 25 Jun 2018 18:07:40 GMT):
should i wait for more than half an hour?
javrevasandeep (Mon, 25 Jun 2018 18:14:24 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=aqFXM4Nd7Gaet64ur) @yacovm yesterday i did the same. I waited for exactly half an hour and then restarted peer1-org1 but still could able to see these messages popping up again in each peer logs
should i wait for more than half an hour?
javrevasandeep (Mon, 25 Jun 2018 18:23:18 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=GNH5GnZN7pX5Y36ki) Just now i recheck the logs and found something more in the peer1-org1 logs.
https://hastebin.com/dexarihego.hs
mark.zhang (Sat, 30 Jun 2018 02:46:44 GMT):
Has joined the channel.
suchith.arodi (Tue, 03 Jul 2018 18:23:28 GMT):
Has joined the channel.
BabuPallam (Wed, 04 Jul 2018 22:20:53 GMT):
Has joined the channel.
NoLimitHoldem (Wed, 11 Jul 2018 06:17:44 GMT):
Has joined the channel.
jayeshjawale95 (Wed, 11 Jul 2018 07:30:32 GMT):
Has joined the channel.
IgorSim (Thu, 12 Jul 2018 10:59:12 GMT):
hi, one question about service discovery, i checked it (using CLI discover) and it works fine. What i don't understand is the following: fabric client SDK need to have access to TLS certificate of the peer(s) in order to be able to send endorsment proposal to that peer(s). But, i can't seem to find TLS of the peer in any of the query responses that i executed (peer membership, configuration, endorsers). Am i missing something?
yacovm (Thu, 12 Jul 2018 11:05:16 GMT):
> fabric client SDK need to have access to TLS certificate of the peer(s) in order to be able to send endorsment proposal to that peer(s)
of course that's not true.... where did you get that from? @IgorSim
IgorSim (Thu, 12 Jul 2018 11:08:01 GMT):
hm, then what client need in case TLS is enabled on peer ? I though client need CORE_PEER_TLS_CERT_FILE
yacovm (Thu, 12 Jul 2018 11:10:23 GMT):
the client needs its *OWN* TLS certificate
yacovm (Thu, 12 Jul 2018 11:10:48 GMT):
notice that when you use the `discover` command you can mention your TLS certificate, right?
IgorSim (Thu, 12 Jul 2018 11:11:25 GMT):
that's right
yacovm (Thu, 12 Jul 2018 11:11:31 GMT):
``--tlsKey=`
yacovm (Thu, 12 Jul 2018 11:11:31 GMT):
`--tlsKey=`
yacovm (Thu, 12 Jul 2018 11:11:41 GMT):
and `--tlsCert`
yacovm (Thu, 12 Jul 2018 11:11:48 GMT):
but you don't have to... right?
yacovm (Thu, 12 Jul 2018 11:12:04 GMT):
that's because if you don't put - the `discover` command in the background generates one for you
yacovm (Thu, 12 Jul 2018 11:12:08 GMT):
and uses that
yacovm (Thu, 12 Jul 2018 11:12:22 GMT):
but you don't need the TLS of the peer, obviously
IgorSim (Thu, 12 Jul 2018 11:15:10 GMT):
if i understand right, tlsKey and tlsCert are used only if TLS client authentication is turned ON at peer side, is that corect?
IgorSim (Thu, 12 Jul 2018 11:15:10 GMT):
if i understand right, tlsKey and tlsCert should be used only if TLS client authentication is turned ON at peer side, is that corect?
yacovm (Thu, 12 Jul 2018 11:26:58 GMT):
yep @IgorSim
yacovm (Thu, 12 Jul 2018 11:27:22 GMT):
but... service discovery enforces client TLS authentication at the application level
yacovm (Thu, 12 Jul 2018 11:27:45 GMT):
but the `discover` CLI - creates a TLS certificate for you, if you don't specify one in the config
IgorSim (Thu, 12 Jul 2018 11:40:02 GMT):
yes, tnx, i see what you just explained in the documentation, that is clear
IgorSim (Thu, 12 Jul 2018 13:37:06 GMT):
going back to my question, i will try to explain it better. Maybe i should ask in sdk-java channel, but anyway here it is: When channel is initialized and peer is added to channel you can specify few peer properties .
One of them is called: 'pemFile' - File location for x509 pem certificate for SSL
I thought this property basically represents CORE_PEER_TLS_CERT_FILE, if not then what is it?
yacovm (Thu, 12 Jul 2018 13:37:58 GMT):
@IgorSim that is a *peer* configuration
yacovm (Thu, 12 Jul 2018 13:38:02 GMT):
not a *client* config ;)
IgorSim (Thu, 12 Jul 2018 13:49:03 GMT):
that's right, CORE_PEER_TLS_CERT_FILE is indeed peer configuration but does client, i guess client==SDK, right :) , need that certificate during channel initialization, i.e. when adding a peer ?
yacovm (Thu, 12 Jul 2018 13:55:17 GMT):
I don't understand what you're asking
IgorSim (Thu, 12 Jul 2018 14:10:04 GMT):
Let's say i'm using Fabric Java SDK, that means in order to be able to transact with HLF from Java SDK i need to initialize channel and add endorsing peer(s) and orderer to the channel.
API looks like this, for example:
Channel channel = client.newChannel(
IgorSim (Thu, 12 Jul 2018 14:10:04 GMT):
Let's say i'm using Fabric Java SDK, that means in order to be able to transact with HLF from Java SDK i need to initialize channel and add endorsing peer(s) and orderer to the channel.
API looks like this, for example:
Channel channel = client.newChannel(
yacovm (Thu, 12 Jul 2018 14:12:35 GMT):
oh
yacovm (Thu, 12 Jul 2018 14:12:44 GMT):
so I think it's the *CA certificate* of the peer ;)
yacovm (Thu, 12 Jul 2018 14:12:44 GMT):
so I think it's the *TLS CA certificate* of the peer ;)
yacovm (Thu, 12 Jul 2018 14:13:15 GMT):
but let's ask @rickr ... he would know for sure
rickr (Thu, 12 Jul 2018 14:24:10 GMT):
https://github.com/hyperledger/fabric-sdk-java/blob/c5c31b8100f62a146220cffb6d304921dfce5829/src/main/java/org/hyperledger/fabric/sdk/HFClient.java#L286-L297
trust certificates
IgorSim (Thu, 12 Jul 2018 14:31:52 GMT):
@rickr yeah, i see, thanks, let's say peer doesn't have mutual TLS enabled, that means 'clientKeyFile' and 'clientCertFile' can be skipped. I'm not clear about 'pemFile' and unfortunately java doc doesn't help me as well. If for example i set 'pemFile' to CORE_PEER_TLS_CERT_FILE then SDK works fine.
But, is that right setting, that is what i'm trying to understand...... I guess not because as far i understood discovery service doesn't return that value hence i can't 'dynamically' add and configure all peers in the channel
yacovm (Thu, 12 Jul 2018 14:36:16 GMT):
how do you know that discovery doesn't return that value @IgorSim ?
IgorSim (Thu, 12 Jul 2018 14:38:28 GMT):
well, i executed supported server-side queries but i don't see it in the response, maybe i'm missing something...that lead me to ask myself question, what is pemFile, what does it represents
IgorSim (Thu, 12 Jul 2018 14:39:15 GMT):
so far, when i used SDK, i set pemFile=CORE_PEER_TLS_CERT_FILE
IgorSim (Thu, 12 Jul 2018 14:39:25 GMT):
does that make sense? :)
rickr (Thu, 12 Jul 2018 14:42:25 GMT):
If the peer your doing discovery on is TLS enabled you're going to have provide a trust chain for that server certificate (the peer) that's what the pemBytes and pemFile are for. You'll also need the client* if the server (peer) is enforcing mutual TLS
IgorSim (Fri, 13 Jul 2018 05:17:05 GMT):
so basically, executing discovery configuration query will return the trust chain under: msps/
yacovm (Fri, 13 Jul 2018 07:10:27 GMT):
yeah... it returns all the MSP config @IgorSim
IgorSim (Fri, 13 Jul 2018 09:11:54 GMT):
ok, tnx
louisliu2048 (Wed, 18 Jul 2018 02:20:11 GMT):
Has joined the channel.
IsaacWong (Wed, 18 Jul 2018 04:54:04 GMT):
Has joined the channel.
braduf (Tue, 24 Jul 2018 21:05:37 GMT):
Has joined the channel.
javrevasandeep (Wed, 25 Jul 2018 06:24:01 GMT):
@yacovm @IgorSim I am facing issue while doing the invoke. It seems to be VSCC endorsement failure kind of error. Below given are the peer0Org1 logs
2018-07-25 05:59:45.983 UTC [shim] func1 -> DEBU 74d [0ebe1dea]Received message TRANSACTION from shim
2018-07-25 05:59:45.983 UTC [shim] handleMessage -> DEBU 74e [0ebe1dea]Handling ChaincodeMessage of type: TRANSACTION(state:ready)
2018-07-25 05:59:45.984 UTC [shim] beforeTransaction -> DEBU 74f [0ebe1dea]Received TRANSACTION, invoking transaction on chaincode(Src:ready, Dst:ready)
2018-07-25 05:59:45.984 UTC [vscc] Invoke -> DEBU 750 VSCC invoked
2018-07-25 05:59:45.984 UTC [vscc] deduplicateIdentity -> DEBU 751 Signature set is of size 1 out of 1 endorsement(s)
2018-07-25 05:59:45.984 UTC [vscc] Invoke -> WARN 752 Endorsement policy failure for transaction txid=bbae754cdb42ccf121ecb77eecfd51190c618416ffabc59dcce47f02693957a9, err: signature set did not satisfy policy
2018-07-25 05:59:45.984 UTC [shim] func1 -> DEBU 753 [0ebe1dea]Transaction completed. Sending COMPLETED
2018-07-25 05:59:45.984 UTC [shim] func1 -> DEBU 754 [0ebe1dea]Move state message COMPLETED
2018-07-25 05:59:45.985 UTC [shim] handleMessage -> DEBU 755 [0ebe1dea]Handling ChaincodeMessage of type: COMPLETED(state:ready)
2018-07-25 05:59:45.985 UTC [shim] func1 -> DEBU 756 [0ebe1dea]send state message COMPLETED
2018-07-25 05:59:45.985 UTC [chaincode] processStream -> DEBU 757 [0ebe1dea]Received message COMPLETED from shim
2018-07-25 05:59:45.985 UTC [chaincode] handleMessage -> DEBU 758 [0ebe1dea]Fabric side Handling ChaincodeMessage of type: COMPLETED in state ready
2018-07-25 05:59:45.985 UTC [chaincode] handleMessage -> DEBU 759 [0ebe1dea-fd85-43b1-a291-b5aa50093a48]HandleMessage- COMPLETED. Notify
2018-07-25 05:59:45.985 UTC [chaincode] notify -> DEBU 75a notifying Txid:0ebe1dea-fd85-43b1-a291-b5aa50093a48, channelID:mychannel
2018-07-25 05:59:45.985 UTC [chaincode] Execute -> DEBU 75b Exit
2018-07-25 05:59:45.985 UTC [lockbasedtxmgr] Done -> DEBU 75c Done with transaction simulation / query execution [bbae754cdb42ccf121ecb77eecfd51190c618416ffabc59dcce47f02693957a9]
2018-07-25 05:59:45.985 UTC [committer/txvalidator] VSCCValidateTxForCC -> DEBU 75d VSCCValidateTxForCC completes for envbytes 0xc421e98800
2018-07-25 05:59:45.985 UTC [committer/txvalidator] VSCCValidateTx -> DEBU 75e VSCCValidateTx completes for env 0xc423166e10 envbytes 0xc421e98800
2018-07-25 05:59:45.985 UTC [committer/txvalidator] validateTx -> ERRO 75f VSCCValidateTx for transaction txId = bbae754cdb42ccf121ecb77eecfd51190c618416ffabc59dcce47f02693957a9 returned error: VSCC error: endorsement policy failure, err: signature set did not satisfy policy
2018-07-25 05:59:45.985 UTC [committer/txvalidator] validateTx -> DEBU 760 validateTx completes for block 0xc42359a640 env 0xc423166e10 txn 0
2018-07-25 05:59:45.986 UTC [committer/txvalidator] Validate -> DEBU 761 got result for idx 0, code 10
2018-07-25 05:59:45.986 UTC [committer/txvalidator] Validate -> DEBU 762 END Block Validation
2018-07-25 05:59:45.987 UTC [kvledger] CommitWithPvtData -> DEBU 763 Channel [mychannel]: Validating state for block [4]
2018-07-25 05:59:45.987 UTC [lockbasedtxmgr] ValidateAndPrepare -> DEBU 764 Validating new block with num trans = [1]
2018-07-25 05:59:45.987 UTC [valimpl] ValidateAndPrepareBatch -> DEBU 765 ValidateAndPrepareBatch() for block number = [4]
2018-07-25 05:59:45.987 UTC [valimpl] ValidateAndPrepareBatch -> DEBU 766 preprocessing ProtoBlock...
2018-07-25 05:59:45.987 UTC [valimpl] preprocessProtoBlock -> WARN 767 Channel [mychannel]: Block [4] Transaction index [0] TxId [bbae754cdb42ccf121ecb77eecfd51190c618416ffabc59dcce47f02693957a9] marked as invalid by committer. Reason code [ENDORSEMENT_POLICY_FAILURE]
yacovm (Wed, 25 Jul 2018 06:26:12 GMT):
@javrevasandeep is this gossip related?
zmaro (Fri, 03 Aug 2018 15:21:03 GMT):
Has joined the channel.
dvliman (Sat, 04 Aug 2018 02:40:47 GMT):
Has joined the channel.
akshay.sood (Sat, 04 Aug 2018 06:47:10 GMT):
Has joined the channel.
1234 (Sat, 18 Aug 2018 14:52:05 GMT):
Has joined the channel.
chandrika (Sun, 19 Aug 2018 12:22:35 GMT):
Has joined the channel.
chandrika (Sun, 19 Aug 2018 12:31:51 GMT):
Hi Expert, I am currently using Fabric 1.2 and have setup my network in 3 ubuntu servers each with 1 org, 1couch, 1 ca, 1 cli, 1 orderer, 1 kakfa and 1 zookeeper..
my channel creation is also working and i can install the chaincodes in all the 3 peers
instantiate the chaincode in one of the peers...
Then i am trying to leverage the private data collection feature of Fabric 1.2. while doing so, i observed when i am firing the invoke command, i am getting an error in my 2nd peer's log whcih says - "Do not know any peer in the channel( mychannel ) that matches the policies , aborting
[gossip/privdata] fetchFromPeers -> WARN 494 Failed fetching private data for block 2 from peers: Empty membership"
due to which when i am trying to query the data in my second peer who has the access to the private data, i am not getting the same and an error is thrown again which says - "Error: endorsement failure during query. response: status:500 message:"{\"Error\":\"Failed to get state for marble1\"}"
"
in my peer log i found this eeror - " HandleTransaction -> ERRO 6eb [4d08c75d] Failed to handle GET_STATE. error: Private data matching public hash version is not available. Public hash version = &version.Height{BlockNum:0x2, TxNum:0x0}, Private data version = (*version.Height)(nil)"
chandrika (Sun, 19 Aug 2018 12:31:51 GMT):
. I took the same network which you shared. Only difference which i did is - I am using network_mode=host and trying to setup the network in 3 separate vms. Rest of the configurations i didnt change really... The error is saying something like this - "2018-08-18 09:10:22.837 UTC [gossip/privdata] fetch -> WARN 534 Do not know any peer in the channel( mychannel ) that matches the policies , aborting
2018-08-18 09:10:22.837 UTC [gossip/privdata] fetchFromPeers -> WARN 535 Failed fetching private data for block 2 from peers: Empty membership
"
chandrika (Sun, 19 Aug 2018 12:31:51 GMT):
Can someone please help me with the correct gossip configuration on each of these vms?
chandrika (Sun, 19 Aug 2018 12:31:51 GMT):
Hi Expert, I am currently using Fabric 1.2 and have setup my network in 3 ubuntu servers each with 1 org, 1couch, 1 ca, 1 cli, 1 orderer, 1 kakfa and 1 zookeeper..
my channel creation is also working and i can install the chaincodes in all the 3 peers
instantiate the chaincode in one of the peers...
Then i am trying to leverage the private data collection feature of Fabric 1.2. while doing so, i observed when i am firing the invoke command, i am getting an error in my 2nd peer's log whcih says - "Do not know any peer in the channel( mychannel ) that matches the policies , aborting
[gossip/privdata] fetchFromPeers -> WARN 494 Failed fetching private data for block 2 from peers: Empty membership"
due to which when i am trying to query the data in my second peer who has the access to the private data, i am not getting the same and an error is thrown again which says - "Error: endorsement failure during query. response: status:500 message:"{\"Error\":\"Failed to get state for marble1\"}"
"
in my peer log i found this eeror - " HandleTransaction -> ERRO 6eb [4d08c75d] Failed to handle GET_STATE. error: Private data matching public hash version is not available. Public hash version = &version.Height{BlockNum:0x2, TxNum:0x0}, Private data version = (*version.Height)(nil)"
. I took the same network which you shared. Only difference which i did is - I am using network_mode=host and trying to setup the network in 3 separate vms. Rest of the configurations i didnt change really... The error is saying something like this - "2018-08-18 09:10:22.837 UTC [gossip/privdata] fetch -> WARN 534 Do not know any peer in the channel( mychannel ) that matches the policies , aborting
2018-08-18 09:10:22.837 UTC [gossip/privdata] fetchFromPeers -> WARN 535 Failed fetching private data for block 2 from peers: Empty membership
yacovm (Sun, 19 Aug 2018 13:13:15 GMT):
https://hyperledger-fabric.readthedocs.io/en/latest/discovery-cli.html
yacovm (Sun, 19 Aug 2018 13:13:17 GMT):
use this please
yacovm (Sun, 19 Aug 2018 13:13:28 GMT):
specifically - `discover peers`
yacovm (Sun, 19 Aug 2018 13:13:34 GMT):
download it from https://nexus.hyperledger.org/content/repositories/releases/org/hyperledger/fabric/hyperledger-fabric/
yacovm (Sun, 19 Aug 2018 13:13:40 GMT):
and then tell me if your peers see other peers
moyating (Mon, 20 Aug 2018 03:30:55 GMT):
Has joined the channel.
rsherwood (Mon, 20 Aug 2018 09:05:04 GMT):
In our current 1.0 configuration we don't have any anchor peer definitions ( we wanted our recovery to go back to the orderer and we only have one peer per organisation) . I see that read the docs for the service discovery says "and uses the network metadata information maintained by the gossip communication layer to find out which peers are online. " If we move to 1.2 will the service discovery service work correctly if we don't have anchor peers defined ?
yacovm (Mon, 20 Aug 2018 09:50:07 GMT):
it will only work for the peer's organization @rsherwood
yacovm (Mon, 20 Aug 2018 09:50:13 GMT):
and the peers won't know about other peers
rsherwood (Mon, 20 Aug 2018 10:16:14 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=ghXca3aLCjGY8Wrvv) @yacovm How much of the service discovery will not work ? It says "With service discovery, applications no longer need to specify which peers they need endorsements from. The SDK can simply send a query to the discovery service asking which peers are needed given a channel and a chaincode ID. The discovery service will then compute a descriptor comprised of two objects:With service discovery, applications no longer need to specify which peers they need endorsements from. The SDK can simply send a query to the discovery service asking which peers are needed given a channel and a chaincode ID. The discovery service will then compute a descriptor comprised of two objects:" Will the discovery service correctly identify the peers in other organisations, but not which are up to date , or will more that this not work ?
yacovm (Mon, 20 Aug 2018 10:21:31 GMT):
so, endorsement policies that require AND from some organizations
yacovm (Mon, 20 Aug 2018 10:21:33 GMT):
won't work
rsherwood (Mon, 20 Aug 2018 10:34:23 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=HAS8kDqAWX7t69S2i) @yacovm Given that most of endorsement policy's have an "AND" between organization , I think you are saying that the "Endorsement query" will not work . What about the "Peer membership query" I assume that will not work either ?
yacovm (Mon, 20 Aug 2018 10:43:41 GMT):
it will work
yacovm (Mon, 20 Aug 2018 10:44:10 GMT):
look, not having anchor peers also makes you unable to use private data for inter-organizations
rsherwood (Mon, 20 Aug 2018 10:55:36 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=ibM66QxhiQS8RuZEa) @yacovm Hi , right now we don't use the private member data, although I can see a way it would be useful to us . We are planning our upgrade to 1.2 and I'm just trying to workout what we need to change, and what will work given that we don't want to make too many application changes, given there is limited budget.
yacovm (Mon, 20 Aug 2018 10:59:18 GMT):
you need to add anchor peers
yacovm (Mon, 20 Aug 2018 10:59:41 GMT):
> given there is limited budget.
I can give you a 100% discount and you can have anchor peers for free
yacovm (Mon, 20 Aug 2018 10:59:41 GMT):
> given there is limited budget.
I can give you a 100% discount and you can have anchor peers for free
yacovm (Mon, 20 Aug 2018 10:59:44 GMT):
how does that sound?
yacovm (Mon, 20 Aug 2018 11:00:19 GMT):
if you don't tend to constantly add and remove peers from the channel you can just make them all anchor peers
C0rWin (Mon, 20 Aug 2018 11:29:36 GMT):
chandrika (Mon, 20 Aug 2018 12:27:32 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=MigSoPDsYzdg2rt2v) @yacovm Thanks @yacovm for your suggestion, I followed your steps and found that my 1st peer is unable to discover other peers. No other MSPs are being shown in the output json after i fire the discover command for peers. Can you please suggest what can i do to make other peers discoverable?
yacovm (Mon, 20 Aug 2018 12:28:09 GMT):
did you join it to the channel?
yacovm (Mon, 20 Aug 2018 12:28:13 GMT):
do you have anchor peers?
chandrika (Mon, 20 Aug 2018 12:29:21 GMT):
yes. I did join all the peers in the channel. and i have the follwoing anchor peer configuration in my configtx.yaml
yacovm (Mon, 20 Aug 2018 12:29:56 GMT):
you need to issue a config update for anchor peers separately
yacovm (Mon, 20 Aug 2018 12:29:59 GMT):
not in the genesis block
yacovm (Mon, 20 Aug 2018 12:30:04 GMT):
after the channel is created
chandrika (Mon, 20 Aug 2018 12:30:41 GMT):
how to do that? any help? any suggestion?
yacovm (Mon, 20 Aug 2018 12:30:46 GMT):
yes
yacovm (Mon, 20 Aug 2018 12:31:24 GMT):
https://github.com/hyperledger/fabric/blob/release-1.2/examples/e2e_cli/scripts/script.sh#L110-L125
chandrika (Mon, 20 Aug 2018 12:32:05 GMT):
ok thanks .. let me check
chandrika (Mon, 20 Aug 2018 12:33:56 GMT):
do i have to update the channel with anchor peers after its creation?
chandrika (Mon, 20 Aug 2018 12:38:33 GMT):
Thanks a lot @yacovm .. its working now... :)
OviiyaDominic (Tue, 21 Aug 2018 07:11:37 GMT):
Has joined the channel.
bdjidi (Tue, 21 Aug 2018 22:48:04 GMT):
Has joined the channel.
OviiyaDominic (Wed, 22 Aug 2018 04:35:27 GMT):
Hi , I have setup my network with 6 orgs ; two peers each , 1 orderer , 6 CA's and 2 channels. During channel creation and join i get an orderer log stating "identity 0 does not satisfy principal: the identity is a member of a different MSP ... " . Can someone help me with the solution for this error?
OviiyaDominic (Wed, 22 Aug 2018 04:36:38 GMT):
anchor peer update .png
1234 (Wed, 22 Aug 2018 04:37:07 GMT):
2-0
nukulsharma (Fri, 24 Aug 2018 09:06:30 GMT):
Has joined the channel.
nukulsharma (Fri, 24 Aug 2018 09:06:54 GMT):
Hi
I have couple of queries -
1. Private Data collections - I could not find private data history , GetHistoryForKey - returns only public data
Note: i am using public data as normal ledger , whilst only private data as Collections
2. Is there any way i can find say recent 10 blocks from ledger, i dont see any API for it. The one we have is getBlockByTxID , which become quite tedious , since first we need to invoke transaction history and then invoke for each one of them to get Block . Any simpler way to this ?
1234 (Fri, 24 Aug 2018 09:14:58 GMT):
@OviiyaDominic
gormand (Fri, 24 Aug 2018 13:42:25 GMT):
Hi - I'm trying to get the discover command to work. I've setup BYFN (which has GOSSIP running). I've created the config file using the cmd discover (with private key and cert of Admin and TLSCA cert). When I run the following command from within the fabricsamples/bin directory I get an error:
$ ../bin/discover --configFile org1conf.yaml peers --channel mychannel --server peer0.org1.example.com:7051
[failed connecting to peer0.org1.example.com:7051: failed connecting to discovery service: failed to create new connection: context deadline exceeded]
I've tried multiple host/port combinations and checked the keys/cert in the configfile. Any suggestions on how to debug this please? Thanks!
Sileymane (Fri, 24 Aug 2018 16:52:43 GMT):
Has joined the channel.
yacovm (Fri, 24 Aug 2018 18:30:17 GMT):
@gormand did you run this from the CLI container?
yacovm (Fri, 24 Aug 2018 18:30:30 GMT):
The address needs to be resolvable
gormand (Sun, 26 Aug 2018 15:31:23 GMT):
Hi @yacovm , On my mac I ran it from the /bin directory after installing FabricSamples. For the peer URL I've used 0.0.0.0:7051. I can see this command is making connection to the peer as I've turned on gRPC trace on the peer and can see:
grpc: Server.Serve failed to complete security handshake from "192.168.192.1:44144": remote error: tls: bad certificate
I've tried various combinations of certificates but can't seem to resolve this problem.
My config file looks like:
daves-mbp-2:first-network gormand$ cat org1conf.yaml
version: 0
tlsconfig:
certpath: ""
keypath: ""
peercacertpath: /Users/blockchain/fabric-samples/fabric-samples/first-network/crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt
timeout: 0s
signerconfig:
mspid: Org1MSP
identitypath: /Users/blockchain/fabric-samples/fabric-samples/first-network/crypto-config/peerOrganizations/org1.example.com/users/Admin@org1.example.com/msp/signcerts/Admin@org1.example.com-cert.pem
keypath: /Users/blockchain/fabric-samples/fabric-samples/first-network/crypto-config/peerOrganizations/org1.example.com/users/Admin@org1.example.com/msp/keystore/c273401c5cbcaaa618950b2be4410ec6b640611dfd979870259c9f258547237f_sk
The command looks like:
../bin/discover --configFile ./org1conf.yaml peers --channel mychannel --server 0.0.0.0:7051
Thanks for your help!
yacovm (Sun, 26 Aug 2018 16:32:12 GMT):
no... don't do that
yacovm (Sun, 26 Aug 2018 16:32:16 GMT):
don't use 0.0.0.0
yacovm (Sun, 26 Aug 2018 16:32:31 GMT):
let's make an experiment
yacovm (Sun, 26 Aug 2018 16:32:44 GMT):
add to your `/etc/hosts` the peer as 127.0.0.1
yacovm (Sun, 26 Aug 2018 16:32:55 GMT):
and then use `--server 127.0.0.1:7051` @gormand
qiangjiyi (Tue, 28 Aug 2018 02:10:36 GMT):
Has joined the channel.
gormand (Tue, 28 Aug 2018 10:44:13 GMT):
Hi @yacovm , Some good news. Within in the *CLI* container I installed the *fabric-samples bin* directory to get the *discover* command and then ran the command successfully using the same privkey and certs as above. As I was in the CLI container on the same docker network as the peers I specified the host url as *peer0.org1.example.com:7051*. It responded in an instant.
I've still not been able to get the discover command working from my host mac to the peer running in the docker container though :( I checked */etc/hosts* and it has *127.0.0.1 localhost* so I specified *127.0.0.1:7051* as the url but I still get the *tls bad certificate* message in the peer logs.
yacovm (Tue, 28 Aug 2018 10:46:07 GMT):
no, you need to put only the IP address @gormand
yacovm (Tue, 28 Aug 2018 10:46:10 GMT):
not the entire URL....
yacovm (Tue, 28 Aug 2018 10:46:38 GMT):
associate `127.0.0.1` with `peer0.org1.example.com`
yacovm (Tue, 28 Aug 2018 10:46:41 GMT):
in the `/etc/hosts`
gormand (Tue, 28 Aug 2018 11:33:52 GMT):
Hi @yacovm , that worked!! Adding `127.0.0.1 peer0.org1.example.com` to the hosts file has fixed the connection issue. Do you know why this is needed?
yacovm (Tue, 28 Aug 2018 11:37:44 GMT):
because when you do the TLS handshake
yacovm (Tue, 28 Aug 2018 11:37:51 GMT):
the remote peer presents its certificate
yacovm (Tue, 28 Aug 2018 11:37:57 GMT):
that has `peer0.org1.example.com`
yacovm (Tue, 28 Aug 2018 11:38:06 GMT):
and you expect 127.0.0.1
yacovm (Tue, 28 Aug 2018 11:38:11 GMT):
so your client denies the connection
gormand (Tue, 28 Aug 2018 11:52:57 GMT):
Hi @yacovm thank you, that makes sense. Thanks for your help!
IgorSim (Wed, 29 Aug 2018 06:34:06 GMT):
hi, i have one q, not sure if its related to gossip protocol, anyway here it is. I have simple network that contains 3 ORGs, 2 of them are having one peer, 3rd ORG is owning an ordering node(solo). Each organization peer is also an anchor peer and leading peer. Network was working fine for few days/weeks, but yesterday i noticed that one of the peers was missing few blocks. I checked orderer logs and saw that missing block(s) were delivered to both peers. I guess these are the messages indicating the delivery:
[common/deliver] deliverBlocks -> DEBU 1587^[[0m [channel: xx] Delivering block for (..) for peer1Org1:52502
[common/deliver] deliverBlocks -> DEBU 1589^[[0m [channel: xx] Delivering block for (..) for peer1Org2:33476
Next, i check peer logs, i noticed messages like the one below doesn't exist on peer which was missing few blocks
[blocksProvider] DeliverBlocks -> DEBU 9d9000^[[0m [xx] Adding payload locally, buffer seqNum = [33], peers number [1]
[blocksProvider] DeliverBlocks -> DEBU 9d9001^[[0m [xx] Gossiping block [33], peers number [1]
Does it mean block never reached the peer? Does orderer have some kind of re-try in this case? I didn't see anything in the logs.
I was also wondering if somehow problematic peer will understand that its own ledger height is not up-to-date(through gossip message or other kind of message coming from peer from the other ORG) and will try to 'pull' the missing blocks, or that's not how it works by design?
Btw, when i restarted peer container, peer received missing blocks. It looks like peer was 'stuck' for some reason or there was network issue between orderer <->peer.
One more thing, are gossip protocol functionalities (sync ledger data for ex)relevant only for peers from same org or also for cross-org?
IgorSim (Wed, 29 Aug 2018 06:34:06 GMT):
hi, i have one q, not sure if its related to gossip protocol, anyway here it is. I have simple network that contains 3 ORGs, 2 of them are having one peer, 3rd ORG is owning an ordering node(solo). Each organization peer is also an anchor peer and leading peer. Network was working fine for few days/weeks, but yesterday i noticed that one of the peers was missing few blocks. I checked orderer logs and saw that missing block(s) were delivered to both peers. I guess these are the messages indicating the delivery:
[common/deliver] deliverBlocks -> DEBU 1587^[[0m [channel: xx] Delivering block for (..) for peer1Org1:52502
[common/deliver] deliverBlocks -> DEBU 1589^[[0m [channel: xx] Delivering block for (..) for peer1Org2:33476
Next, i check peer logs, i noticed messages like the one below doesn't exist on peer which was missing few blocks
[blocksProvider] DeliverBlocks -> DEBU 9d9000^[[0m [xx] Adding payload locally, buffer seqNum = [33], peers number [1]
[blocksProvider] DeliverBlocks -> DEBU 9d9001^[[0m [xx] Gossiping block [33], peers number [1]
Does it mean block never reached the peer? Does orderer have some kind of re-try in this case? I didn't see anything in the logs.
I was also wondering if somehow problematic peer will understand that its own ledger height is not up-to-date(through gossip message or other kind of message coming from peer from the other ORG) and will try to 'pull' the missing blocks, or that's not how it works by design?
Btw, when i restarted peer container, peer received missing blocks. It looks like peer was 'stuck' for some reason or there was network issue between orderer <->peer.
One more thing, are gossip protocol functionalities (sync ledger data for ex)relevant only for peers from same org or also for cross-org?
IgorSim (Wed, 29 Aug 2018 06:34:06 GMT):
hi, i have one q, not sure if its related to gossip protocol, anyway here it is. I have simple network that contains 3 ORGs, 2 of them are having one peer, 3rd ORG is owning an ordering node(solo). Each organization peer is also an anchor peer and leading peer. Network was working fine for few days/weeks, but yesterday i noticed that one of the peers was missing few blocks. I checked orderer logs and saw that missing block(s) were delivered to both peers. I guess these are the messages indicating the delivery:
[common/deliver] deliverBlocks -> DEBU 1587^[[0m [channel: xx] Delivering block for (..) for peer1Org1:52502
[common/deliver] deliverBlocks -> DEBU 1589^[[0m [channel: xx] Delivering block for (..) for peer1Org2:33476
Next, i check peer logs, i noticed messages like the one below doesn't exist on peer which was missing few blocks
[blocksProvider] DeliverBlocks -> DEBU 9d9000^[[0m [xx] Adding payload locally, buffer seqNum = [33], peers number [1]
[blocksProvider] DeliverBlocks -> DEBU 9d9001^[[0m [xx] Gossiping block [33], peers number [1]
Does it mean block never reached the peer? Does orderer have some kind of re-try in this case? I didn't see anything in the logs.
I was also wondering if somehow problematic peer will understand that its own ledger height is not up-to-date(through gossip message or other kind of message coming from peer from the other ORG) and will try to 'pull' the missing blocks, or that's not how it works by design?
Btw, when i restarted peer container, peer received missing blocks. It looks like peer was 'stuck' for some reason or there was network issue between orderer <->peer.
One more thing, are gossip protocol functionalities (sync ledger data for ex)relevant only for peers from same org or also for cross-org?
jdfigure (Thu, 30 Aug 2018 20:06:32 GMT):
Has joined the channel.
IgorSim (Sun, 02 Sep 2018 20:16:18 GMT):
Guys, any clue or direction for further investigation will be really appreciated...
yacovm (Sun, 02 Sep 2018 20:22:11 GMT):
oh i missed it
yacovm (Sun, 02 Sep 2018 20:22:13 GMT):
let me read
yacovm (Sun, 02 Sep 2018 20:23:00 GMT):
what do you mean missing a block, Igor?
yacovm (Sun, 02 Sep 2018 20:23:10 GMT):
what version is that?
yacovm (Sun, 02 Sep 2018 20:23:10 GMT):
what version of fabric is that?
yacovm (Sun, 02 Sep 2018 20:23:31 GMT):
and how can we investigate now when you restarted the peer??? :facepalm:
yacovm (Sun, 02 Sep 2018 20:23:31 GMT):
and how can we investigate now when you restarted the peer??? @IgorSim
yacovm (Sun, 02 Sep 2018 20:24:07 GMT):
when someone dies in a crime scene the policy always says not to move the body....
yacovm (Sun, 02 Sep 2018 20:24:07 GMT):
when someone dies in a crime scene the police always says not to move the body....
yacovm (Sun, 02 Sep 2018 20:30:53 GMT):
If it happens again please tell and tag me
IgorSim (Sun, 02 Sep 2018 20:36:46 GMT):
i have all the logs
IgorSim (Sun, 02 Sep 2018 20:37:06 GMT):
fabric 1.2
IgorSim (Sun, 02 Sep 2018 20:39:53 GMT):
@yacovm 'missing a block'..i mean peer from the second ORG was missing few blocks, although in orderer logs i saw that ordered is delivering the blocks to that peer, have no idea if they reached, it looks like they didn't
IgorSim (Sun, 02 Sep 2018 20:39:53 GMT):
@yacovm 'missing a block'..i mean peer from the second ORG was missing few blocks, although in orderer logs i saw that orderer is delivering the blocks to that peer, have no idea if they reached, it looks like they didn't
yacovm (Sun, 02 Sep 2018 20:41:42 GMT):
what do you mean missing a block? did he get the blocks after?
yacovm (Sun, 02 Sep 2018 20:41:46 GMT):
@IgorSim
IgorSim (Sun, 02 Sep 2018 20:41:54 GMT):
only after restart
yacovm (Sun, 02 Sep 2018 20:42:18 GMT):
smells to me like a ledger deadlock. We had a bug fixed in v1.2
yacovm (Sun, 02 Sep 2018 20:42:28 GMT):
that involves this kind of thing
yacovm (Sun, 02 Sep 2018 20:42:34 GMT):
but I don't remember if a fix was released
yacovm (Sun, 02 Sep 2018 20:43:14 GMT):
oh
IgorSim (Sun, 02 Sep 2018 20:43:17 GMT):
if it was released in 1.2, or planned to be realeased in 1.3 not sure i understand you
yacovm (Sun, 02 Sep 2018 20:43:20 GMT):
right... we didn't release 1.2.1
yacovm (Sun, 02 Sep 2018 20:43:28 GMT):
i think we'll release it either this or next week
IgorSim (Sun, 02 Sep 2018 20:43:33 GMT):
i see
yacovm (Sun, 02 Sep 2018 20:43:40 GMT):
let me show you the bug, hold on
IgorSim (Sun, 02 Sep 2018 20:43:45 GMT):
which bug is this one, can you tell me jira id?
IgorSim (Sun, 02 Sep 2018 20:43:47 GMT):
ok
yacovm (Sun, 02 Sep 2018 20:44:26 GMT):
https://github.com/hyperledger/fabric/commit/912e6a9ad217040e89e03eb5aed7060fbdc25632
yacovm (Sun, 02 Sep 2018 20:44:35 GMT):
FAB-11094
yacovm (Sun, 02 Sep 2018 20:45:17 GMT):
so there was a bug in the ledger that made block commit get stuck
IgorSim (Sun, 02 Sep 2018 20:46:27 GMT):
and that prevent peer from receiving new blocks?
yacovm (Sun, 02 Sep 2018 20:46:56 GMT):
if the ledger is stuck...
yacovm (Sun, 02 Sep 2018 20:47:04 GMT):
obviously it won't commit new blocks
yacovm (Sun, 02 Sep 2018 20:47:11 GMT):
to commit a block you need to obtain a lock
yacovm (Sun, 02 Sep 2018 20:47:19 GMT):
and if the lock is held and isn't released... ;)
yacovm (Sun, 02 Sep 2018 20:47:29 GMT):
but i don't know if this is the case in your problem
yacovm (Sun, 02 Sep 2018 20:47:43 GMT):
did you restart the peer via docker?
IgorSim (Sun, 02 Sep 2018 20:47:47 GMT):
yes
yacovm (Sun, 02 Sep 2018 20:47:50 GMT):
next time if you restart the peer - do something else
yacovm (Sun, 02 Sep 2018 20:47:58 GMT):
`docker exec -it
yacovm (Sun, 02 Sep 2018 20:48:05 GMT):
and then `pkill -SIGABRT peer`
yacovm (Sun, 02 Sep 2018 20:48:16 GMT):
this would make the peer die but also write to stderr the goroutine stack trace
yacovm (Sun, 02 Sep 2018 20:48:25 GMT):
it will show if a deadlock occurred or not
IgorSim (Sun, 02 Sep 2018 20:49:44 GMT):
ok, tnx for the tip....
IgorSim (Mon, 03 Sep 2018 08:33:03 GMT):
@yacovm regarding what we discussed yesterday and related bug....can you please shae few more details regarding 'frequent Deliver cancellations' , what does it mean and when such condition can occurs..thanks
IgorSim (Mon, 03 Sep 2018 08:33:03 GMT):
@yacovm regarding what we discussed yesterday and related bug....can you please share few more details regarding 'frequent Deliver cancellations' , what does it mean and when such condition can occurs..thanks
yacovm (Mon, 03 Sep 2018 08:41:50 GMT):
it's when you use the new event channel service
IgorSim (Wed, 05 Sep 2018 13:08:28 GMT):
@yacovm hi, we encountered the problem again, this time on different environment but symptoms looks the same. Goroutine stack trace can be found at https://pastebin.com/KSXuXNZt ...can you please check and let me know if this is related to the bug
yacovm (Wed, 05 Sep 2018 13:10:00 GMT):
@IgorSim
yacovm (Wed, 05 Sep 2018 13:10:05 GMT):
that's not all the stack trace though
yacovm (Wed, 05 Sep 2018 13:10:22 GMT):
it only has 1 goroutine
IgorSim (Wed, 05 Sep 2018 13:11:28 GMT):
ok, i can paste entire stack trace..i will do it bit later and let you know, tnx
IgorSim (Wed, 05 Sep 2018 14:11:35 GMT):
@yacovm i have entire stack trace, its ~17K lines, 1,7mb, can i upload to some service (for ex uploadfiles.io) and send you download link or you prefer other means?
yacovm (Wed, 05 Sep 2018 14:12:01 GMT):
yeah
yacovm (Wed, 05 Sep 2018 14:12:04 GMT):
upload to the internet
IgorSim (Wed, 05 Sep 2018 14:12:24 GMT):
https://ufile.io/2iyjm
yacovm (Wed, 05 Sep 2018 14:13:30 GMT):
woohoooo
yacovm (Wed, 05 Sep 2018 14:13:35 GMT):
```
goroutine 453 [semacquire, 18 minutes]:
sync.runtime_Semacquire(0xc421fe4fec)
/opt/go/src/runtime/sema.go:56 +0x39
sync.(*RWMutex).RLock(0xc421fe4fe0)
/opt/go/src/sync/rwmutex.go:50 +0x49
github.com/hyperledger/fabric/core/ledger/kvledger.(*kvLedger).GetBlockchainInfo(0xc42201fc70, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/kvledger/kv_ledger.go:177 +0x7c
github.com/hyperledger/fabric/core/committer.(*LedgerCommitter).LedgerHeight(0xc42035e0e0, 0xc422bff0e0, 0x2, 0x2)
/opt/gopath/src/github.com/hyperledger/fabric/core/committer/committer_impl.go:140 +0x37
github.com/hyperledger/fabric/gossip/state.(*GossipStateProviderImpl).addPayload(0xc4203bfc00, 0xc42210cbc0, 0xc42044a701, 0xc42030be60, 0xc422f87d70)
/opt/gopath/src/github.com/hyperledger/fabric/gossip/state/state.go:751 +0xf7
github.com/hyperledger/fabric/gossip/state.(*GossipStateProviderImpl).AddPayload(0xc4203bfc00, 0xc42210cbc0, 0xc42211019c, 0x4)
/opt/gopath/src/github.com/hyperledger/fabric/gossip/state/state.go:739 +0x5a
github.com/hyperledger/fabric/gossip/service.(*gossipServiceImpl).AddPayload(0xc4204b45a0, 0xc42211019c, 0x4, 0xc42210cbc0, 0x0, 0x0)
/opt/gopath/src/github.com/hyperledger/fabric/gossip/service/gossip_service.go:343 +0xba
github.com/hyperledger/fabric/core/deliverservice/blocksprovider.(*blocksProviderImpl).DeliverBlocks(0xc4205ba500)
/opt/gopath/src/github.com/hyperledger/fabric/core/deliverservice/blocksprovider/blocksprovider.go:188 +0xe0d
github.com/hyperledger/fabric/core/deliverservice.(*deliverServiceImpl).launchBlockProvider(0xc42039f830, 0xc42211019c, 0x4, 0xc4205961a0)
/opt/gopath/src/github.com/hyperledger/fabric/core/deliverservice/deliveryclient.go:179 +0x1b8
created by github.com/hyperledger/fabric/core/deliverservice.(*deliverServiceImpl).StartDeliverForChannel
/opt/gopath/src/github.com/hyperledger/fabric/core/deliverservice/deliveryclient.go:166 +0x601
```
yacovm (Wed, 05 Sep 2018 14:13:58 GMT):
so
yacovm (Wed, 05 Sep 2018 14:14:09 GMT):
I think it's that `Deliver` bug which I talked about
yacovm (Wed, 05 Sep 2018 14:14:15 GMT):
@IgorSim
yacovm (Wed, 05 Sep 2018 14:14:25 GMT):
but let me make sure with Manish
yacovm (Wed, 05 Sep 2018 14:14:42 GMT):
@manish-sethi - the stack trace here... that he uploaded to `https://ufile.io/2iyjm`
manish-sethi (Wed, 05 Sep 2018 14:14:42 GMT):
Has joined the channel.
yacovm (Wed, 05 Sep 2018 14:14:54 GMT):
could you take a look and tell if you think it's that ledger deadlock from back then?
IgorSim (Wed, 05 Sep 2018 14:14:55 GMT):
ok, that will be helpfull, we're setting up production env...this is blocker for us..yo mention it will be released in 1.2.1?
yacovm (Wed, 05 Sep 2018 14:15:01 GMT):
he's running v1.2.0
yacovm (Wed, 05 Sep 2018 14:15:14 GMT):
can you perhaps apply a local patch to your environment?
yacovm (Wed, 05 Sep 2018 14:15:20 GMT):
I can just give you a git diff
IgorSim (Wed, 05 Sep 2018 14:15:53 GMT):
and apply the change on the peer ?
yacovm (Wed, 05 Sep 2018 14:15:57 GMT):
yes
IgorSim (Wed, 05 Sep 2018 14:16:05 GMT):
ok, pls give me that
yacovm (Wed, 05 Sep 2018 14:16:22 GMT):
it's the stuff from https://github.com/hyperledger/fabric/commit/912e6a9ad217040e89e03eb5aed7060fbdc25632
IgorSim (Wed, 05 Sep 2018 14:16:27 GMT):
i see
IgorSim (Wed, 05 Sep 2018 14:16:57 GMT):
btw, are there plans for 1.2.1 or next release is 1.3?
yacovm (Wed, 05 Sep 2018 14:17:01 GMT):
https://gerrit.hyperledger.org/r/changes/24217/revisions/912e6a9ad217040e89e03eb5aed7060fbdc25632/patch?zip
yacovm (Wed, 05 Sep 2018 14:17:04 GMT):
download this
yacovm (Wed, 05 Sep 2018 14:17:08 GMT):
this is a git patch
yacovm (Wed, 05 Sep 2018 14:17:28 GMT):
> btw, are there plans for 1.2.1 or next release is 1.3?
good question.... @mastersingh24 , @IgorSim is waiting for v1.2.1 what is the ETA?
IgorSim (Wed, 05 Sep 2018 14:19:56 GMT):
thank you
manish-sethi (Wed, 05 Sep 2018 14:35:52 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=Xt8qXyJbtakcoCPSX) @yacovm Yes, this is the same issue...
IgorSim (Fri, 07 Sep 2018 07:51:30 GMT):
@yacovm , how i can apply the patch on the peer? If i understand right, peer binary is part of the 'fabric-ca' image (under /usr/local/bin/)..does it mean i need to build the peer (get sources, apply patch, build the peer) and then use this new patched version?
yacovm (Fri, 07 Sep 2018 08:02:30 GMT):
you need to build it
yacovm (Fri, 07 Sep 2018 08:02:41 GMT):
how is that part of the fabric-ca ???
yacovm (Fri, 07 Sep 2018 08:03:08 GMT):
you need to clone the fabric repo, - specifically checkout branch `release-1.2`
yacovm (Fri, 07 Sep 2018 08:03:16 GMT):
and then apply the patch and then run `make peer-docker`
IgorSim (Fri, 07 Sep 2018 08:19:58 GMT):
sorry, i meant fabric-ca-peer
IgorSim (Fri, 07 Sep 2018 08:20:55 GMT):
tnx for answer
Ammu (Fri, 07 Sep 2018 10:45:55 GMT):
Error: failed to create deliver client: orderer client failed to connect to orderer.example.com:7050: failed to create new connection: context deadline exceeded"
while creating the channel i am facing the error
yousaf (Sat, 08 Sep 2018 14:33:10 GMT):
Has joined the channel.
yousaf (Sat, 08 Sep 2018 14:52:50 GMT):
@Ammu probably you would have restarted your network. You may have left some old containers running that might be causing conflicts. so first stop and remove the old docker containers then try again.
raviyelleni (Sun, 09 Sep 2018 04:45:28 GMT):
Has joined the channel.
kelvinzhong (Mon, 10 Sep 2018 06:12:28 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=nPn5rJoR2sJE3KX6B) @yacovm hi, i got some similar issue, but is the private data not synchronized, i haven't look into the weather the block height is up-to-date, but one of the peer work fine at first, but later it doesn't update the transaction any more, even restart the peer won't help.
kelvinzhong (Mon, 10 Sep 2018 06:16:14 GMT):
this issue does not happened all the time, but indeed i encounter for few times, i not sure if it's relate to the connection are base on the public network ip, once i change to test in localhost and use the domain inside docker network, seems won't trigger this issue.
kelvinzhong (Mon, 10 Sep 2018 06:16:14 GMT):
this issue does not happened all the time, but indeed i encounter for few times, i'm not sure if it's related to the peer connection are base on the public network ip, once i change to test in localhost and use the domain inside docker network for connection, seems won't trigger this issue.
yacovm (Mon, 10 Sep 2018 07:02:08 GMT):
What similar issue?
yacovm (Mon, 10 Sep 2018 07:02:16 GMT):
Similar to what?
kelvinzhong (Mon, 10 Sep 2018 10:03:14 GMT):
some peer won't synchronized the transaction at some point
yacovm (Mon, 10 Sep 2018 10:03:53 GMT):
can you give some more information please?
kelvinzhong (Mon, 10 Sep 2018 10:03:56 GMT):
i'm not sure is the private data didn't synchronized or the block didn't synchronized
yacovm (Mon, 10 Sep 2018 10:03:58 GMT):
were there any warning in the logs?
yacovm (Mon, 10 Sep 2018 10:04:06 GMT):
well you can check the height of the peer
yacovm (Mon, 10 Sep 2018 10:04:18 GMT):
what version are you using?
kelvinzhong (Mon, 10 Sep 2018 10:04:22 GMT):
1.2
kelvinzhong (Mon, 10 Sep 2018 10:05:19 GMT):
i will try next time, if the peer are connected by public network ip, this would happened
yacovm (Mon, 10 Sep 2018 10:05:23 GMT):
ok then use the discovery CLI to see:
1) Does the peer know other peers
2) The ledger height of the peer
yacovm (Mon, 10 Sep 2018 10:05:32 GMT):
https://hyperledger-fabric.readthedocs.io/en/latest/discovery-cli.html
kelvinzhong (Mon, 10 Sep 2018 10:05:36 GMT):
and always disconnected to the client side
kelvinzhong (Mon, 10 Sep 2018 10:05:55 GMT):
okay i will try next time
kelvinzhong (Mon, 10 Sep 2018 10:06:10 GMT):
it seems all fine in local test
JaydipMakadia (Thu, 13 Sep 2018 13:09:07 GMT):
Has joined the channel.
MeenakshiSingh (Wed, 19 Sep 2018 18:27:51 GMT):
Hi...I am trying to configure cross organization gossip using anchor peers. I have successfully updated the channel configuration with the anchor peer updates on the channel. But the individual peers are not able to communicate with each other. Following are the peer logs:
MeenakshiSingh (Wed, 19 Sep 2018 18:28:21 GMT):
Clipboard - September 19, 2018 11:58 PM
yacovm (Wed, 19 Sep 2018 18:35:04 GMT):
right... they can't connect.
yacovm (Wed, 19 Sep 2018 18:35:09 GMT):
you need to troubleshoot why
MeenakshiSingh (Wed, 19 Sep 2018 18:37:45 GMT):
how do I do that? I am able to normally invoke transactions over the network that means peers are discoverable.
yacovm (Wed, 19 Sep 2018 18:38:55 GMT):
how did you deduce that?
yacovm (Wed, 19 Sep 2018 18:39:07 GMT):
the fact that you can connect to peers, doesn't mean the peers can connect to one another...
yacovm (Wed, 19 Sep 2018 18:39:21 GMT):
you can try to log into the peer's container, or VM
yacovm (Wed, 19 Sep 2018 18:39:35 GMT):
and try to connect from there, to the address it tries (but fails) to connect
MeenakshiSingh (Wed, 19 Sep 2018 18:48:22 GMT):
what protocol does gossip use for communicating? Do I need to enable any other protocol for inbound traffic other than tcp on 7051?
yacovm (Wed, 19 Sep 2018 18:49:40 GMT):
it uses gRPC over TLS
MeenakshiSingh (Wed, 19 Sep 2018 18:50:24 GMT):
any specific configuration required for tls?
MeenakshiSingh (Wed, 19 Sep 2018 18:50:24 GMT):
any specific configuration required for tls at the peers?
yacovm (Wed, 19 Sep 2018 19:03:57 GMT):
no...
MeenakshiSingh (Wed, 19 Sep 2018 19:17:12 GMT):
tested by allowing all traffic, irrespective of protocols. Still no luck. I am able to ping one host from another.
MeenakshiSingh (Wed, 19 Sep 2018 19:18:08 GMT):
any other suggestions?
yacovm (Wed, 19 Sep 2018 19:56:41 GMT):
yes
yacovm (Wed, 19 Sep 2018 19:56:47 GMT):
turn on gRPC logging
yacovm (Wed, 19 Sep 2018 19:57:37 GMT):
https://github.com/hyperledger/fabric/blob/release-1.2/sampleconfig/core.yaml#L50
yacovm (Wed, 19 Sep 2018 19:57:41 GMT):
@MeenakshiSingh
MeenakshiSingh (Thu, 20 Sep 2018 07:37:34 GMT):
Enabled debug level logs - looks like its failing to get a tls handshake. @yacovm - here are the logs - https://pastebin.com/gZ4ASii2 Could you please take a look.
MeenakshiSingh (Thu, 20 Sep 2018 07:37:34 GMT):
Enabled debug level logs - looks like its failing to get a tls handshake. @yacovm - here are the logs - https://pastebin.com/zbXEihcu Could you please take a look.
yacovm (Thu, 20 Sep 2018 08:27:31 GMT):
Let me guess @MeenakshiSingh you didnt add the anchor peer to the channel?
MeenakshiSingh (Thu, 20 Sep 2018 08:33:34 GMT):
I did. All peers are added to the channel - peer channel join command.
MeenakshiSingh (Thu, 20 Sep 2018 08:37:37 GMT):
the network is tls enabled - is that the cause of ```Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"fabric-ca-server\")" ``` Do I need to provide some specific TLS settings?
yacovm (Thu, 20 Sep 2018 08:38:03 GMT):
Lol what is that certificate?
yacovm (Thu, 20 Sep 2018 08:38:26 GMT):
I think you messed up the certificates when you created the channel
MeenakshiSingh (Thu, 20 Sep 2018 08:39:32 GMT):
Umm..messed up? How?
MeenakshiSingh (Thu, 20 Sep 2018 08:42:33 GMT):
Should I regenerate the certificates and bootstrap the network again?
yacovm (Thu, 20 Sep 2018 09:23:00 GMT):
So i dont know what you should do
yousaf (Thu, 20 Sep 2018 11:41:34 GMT):
I am using this command peer channel upgrade and getting this error.......Error: could not assemble transaction, err Proposal response was not successful, error code 500, msg cannot get package for chaincode (mycc:2.0)....any solution?
hyperlearner (Fri, 21 Sep 2018 11:15:29 GMT):
Has joined the channel.
hyperlearner (Fri, 21 Sep 2018 11:17:56 GMT):
hi while trying private data collection in balance transfer i got the following error 'Requested to send to at least 1 peers, but know only of 0 suitable peers
'
what is the relationship between requiredPeerCount and maxPeerCount
dave.enyeart (Fri, 21 Sep 2018 11:20:37 GMT):
read all about the private data policies here: https://hyperledger-fabric.readthedocs.io/en/latest/private-data-arch.html. i asked you to post over here for troubleshooting gossip on balance transfer's network...
C0rWin (Fri, 21 Sep 2018 11:21:24 GMT):
`requiredPeerCount` is states how many peer *at least* private data has to be distributed to
C0rWin (Fri, 21 Sep 2018 11:22:02 GMT):
`maxPeerCount` is the maximum number of peers to sent data to, has to be greater than `requiredPeerCount`
dave.enyeart (Fri, 21 Sep 2018 11:24:07 GMT):
looks like gossip is not configured at all for balance transfer network: https://github.com/hyperledger/fabric-samples/blob/release-1.2/balance-transfer/artifacts/docker-compose.yaml
dave.enyeart (Fri, 21 Sep 2018 11:24:25 GMT):
i'd suggest use first-network sample network, which is pre-configured for gossip and known to support private data
Sreesha (Fri, 21 Sep 2018 11:51:31 GMT):
Has joined the channel.
Sreesha (Fri, 21 Sep 2018 12:17:23 GMT):
Can we integrate fabric-ca examples with balance-transfer
Sreesha (Fri, 21 Sep 2018 12:35:22 GMT):
Iam unable to use certificates generated by fabric-ca to run balance-transfer examples
Sreesha (Fri, 21 Sep 2018 12:35:51 GMT):
During Channel creation it throws error SERVICE UNAVAILABLE
Sreesha (Fri, 21 Sep 2018 12:36:15 GMT):
::::::::::::::::
Sreesha (Fri, 21 Sep 2018 12:36:56 GMT):
===========>Inside Sendbroadcast
E0921 18:09:38.936522004 7808 ssl_transport_security.c:177] ssl_info_callback: error occured.
E0921 18:09:38.936576862 7808 ssl_transport_security.c:921] Handshake failed with fatal error SSL_ERROR_SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate.
error: [Orderer.js]: sendBroadcast - on error: "Error: Connect Failed\n at ClientDuplexStream._emitStatusIfDone (/hyperledger1.1/fabric-samples/balance-transferca/node_modules/grpc/src/node/src/client.js:255:19)\n at ClientDuplexStream._readsDone (/hyperledger1.1/fabric-samples/balance-transferca/node_modules/grpc/src/node/src/client.js:221:8)\n at readCallback (/hyperledger1.1/fabric-samples/balance-transferca/node_modules/grpc/src/node/src/client.js:283:12)"
[2018-09-21 18:09:38.941] [ERROR] Create-Channel - Error: SERVICE_UNAVAILABLE
standrew03 (Sun, 23 Sep 2018 15:44:22 GMT):
Has joined the channel.
VirendraSolanke (Fri, 28 Sep 2018 18:07:59 GMT):
Has joined the channel.
zacpl (Fri, 28 Sep 2018 18:37:04 GMT):
Has joined the channel.
Bartb0 (Tue, 02 Oct 2018 10:52:55 GMT):
Has joined the channel.
Mozer18 (Tue, 02 Oct 2018 18:25:18 GMT):
Has joined the channel.
Mozer18 (Tue, 02 Oct 2018 19:00:11 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=J8JKTKodvbnrJ6CYy) @MeenakshiSingh please check your docker config, i've faced the same error.
MaddaliPadmaja (Wed, 03 Oct 2018 07:14:15 GMT):
Has joined the channel.
mrjdomingus (Thu, 04 Oct 2018 08:11:31 GMT):
Has joined the channel.
qiangqinqq (Sat, 06 Oct 2018 07:35:21 GMT):
Has joined the channel.
bh4rtp (Sun, 07 Oct 2018 07:29:57 GMT):
in the docker-compose.yaml of `fabric-samples/balance-transfer`, `CORE_PEER_GOSSIP_BOOTSTRAP` and `CORE_PEER_GOSSIP_EXTERNALENDPOINT` are set like this:
```peer0.org1.example.com:
container_name: peer0.org1.example.com
environment:
- CORE_PEER_GOSSIP_BOOTSTRAP=peer1.org1.example.com:7051
- CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer0.org1.example.com:7051
ports:
- 7051:7051
- 7053:7053
peer1.org1.example.com:
container_name: peer1.org1.example.com
environment:
- CORE_PEER_GOSSIP_BOOTSTRAP=peer0.org1.example.com:7051
- CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer1.org1.example.com:7051
ports:
- 7056:7051
- 7058:7053```
bh4rtp (Sun, 07 Oct 2018 07:30:20 GMT):
how to set these env for 4 peers?
yacovm (Sun, 07 Oct 2018 12:08:25 GMT):
@bh4rtp what do you mean "for 4 peers" ?
bh4rtp (Sun, 07 Oct 2018 12:17:49 GMT):
@yacovm 4 peers for org1. does `balance-transfer` mistake to configure gossip like above?
bh4rtp (Sun, 07 Oct 2018 12:19:09 GMT):
should `peer0.org1.example.com` remove the `CORE_PEER_GOSSIP_BOOTSTRAP=peer1.org1.example.com:7051`?
yacovm (Sun, 07 Oct 2018 12:20:04 GMT):
no...
yacovm (Sun, 07 Oct 2018 12:20:08 GMT):
why do you think that?
bh4rtp (Sun, 07 Oct 2018 12:22:29 GMT):
i don't understand they are cross configure.
bh4rtp (Sun, 07 Oct 2018 12:22:29 GMT):
i don't understand why they are cross configured.
bh4rtp (Sun, 07 Oct 2018 12:24:48 GMT):
it is different from e2e example in fabric https://github.com/hyperledger/fabric/blob/77c3aa6ce5b0cfba93bfda009095886dbcadff91/examples/e2e_cli/base/docker-compose-base.yaml#L64.
yacovm (Sun, 07 Oct 2018 12:26:10 GMT):
why is it different?
yacovm (Sun, 07 Oct 2018 12:26:18 GMT):
use the configuration in the e2e
yacovm (Sun, 07 Oct 2018 12:26:22 GMT):
it's more up to date
bh4rtp (Sun, 07 Oct 2018 12:29:28 GMT):
so i think `balance-transfer` makes a mistake here.
yacovm (Sun, 07 Oct 2018 12:36:39 GMT):
probably
MikeRichardson (Mon, 08 Oct 2018 16:10:51 GMT):
Has joined the channel.
waxer (Fri, 12 Oct 2018 12:30:05 GMT):
Has joined the channel.
yousaf (Sat, 13 Oct 2018 22:31:19 GMT):
My all docker containers of peers and orderer are exiting except the cli container when i run docker-compose command to up my network. What could be the reason? Any info related to this?
yacovm (Sat, 13 Oct 2018 22:37:59 GMT):
run `docker ps -a` @yousaf and then do `docker logs`
SumanPapanaboina (Sun, 14 Oct 2018 15:36:41 GMT):
Has joined the channel.
Glen (Thu, 18 Oct 2018 12:55:45 GMT):
Has joined the channel.
kh.nguyen (Fri, 19 Oct 2018 00:21:16 GMT):
Has joined the channel.
AndrewNRise (Fri, 19 Oct 2018 08:20:09 GMT):
Has joined the channel.
Ammu (Mon, 22 Oct 2018 07:07:44 GMT):
why decentralization?
Ammu (Mon, 22 Oct 2018 07:07:48 GMT):
in blockchain
Ammu (Mon, 22 Oct 2018 07:08:39 GMT):
all advantages in centralized itself we are getting right?
cagdast (Thu, 25 Oct 2018 07:42:30 GMT):
Has joined the channel.
mrlinjun (Thu, 01 Nov 2018 02:37:14 GMT):
Has joined the channel.
mrlinjun (Thu, 01 Nov 2018 02:39:09 GMT):
Clipboard - November 1, 2018 10:39 AM
mrlinjun (Thu, 01 Nov 2018 02:39:09 GMT):
Clipboard - November 1, 2018 10:39 AM
mrlinjun (Thu, 01 Nov 2018 02:39:09 GMT):
Clipboard - November 1, 2018 10:39 AM
mrlinjun (Thu, 01 Nov 2018 02:39:09 GMT):
Clipboard - November 1, 2018 10:39 AM
mrlinjun (Thu, 01 Nov 2018 02:39:09 GMT):
Clipboard - November 1, 2018 10:39 AM
mrlinjun (Thu, 01 Nov 2018 02:41:58 GMT):
@yacovm
githubcpc (Thu, 01 Nov 2018 09:43:05 GMT):
Has joined the channel.
IgarashiTakashi (Thu, 01 Nov 2018 13:46:44 GMT):
Has joined the channel.
bh4rtp (Sat, 03 Nov 2018 02:31:21 GMT):
@mrlinjun i am facing this warning too. have you solved?
DirkKrueger (Sat, 03 Nov 2018 13:59:28 GMT):
Has joined the channel.
awes0menessInc (Tue, 06 Nov 2018 03:37:27 GMT):
Has joined the channel.
sigma67 (Tue, 06 Nov 2018 08:49:01 GMT):
Has joined the channel.
enriquebusti (Wed, 07 Nov 2018 12:01:21 GMT):
Has joined the channel.
ivanovv (Sat, 10 Nov 2018 21:04:29 GMT):
Has joined the channel.
ivanovv (Sat, 10 Nov 2018 21:15:20 GMT):
Hi, I have similar issue as the hyperlearner above. Went through the docs. I'm using the fabric sdk java. I'm also attaching more info from the logs. Could somebody elaborate on the possible reasons?! Thanks in advance.
ivanovv (Sat, 10 Nov 2018 21:16:37 GMT):
privateCollectionIssue.txt
AlexanderZhovnuvaty (Mon, 12 Nov 2018 11:05:54 GMT):
Has left the channel.
maozhuzi (Mon, 12 Nov 2018 13:47:47 GMT):
Has joined the channel.
h4995974 (Thu, 15 Nov 2018 22:03:21 GMT):
Has joined the channel.
h4995974 (Thu, 15 Nov 2018 22:03:22 GMT):
Hello everyone.
We're receiving the following error when trying to spin up peers inside a kubernetes pods. `2018-11-15 21:52:25.172 UTC [gossip/gossip] handleMessage -> WARN 01f Message GossipMessage: tag:EMPTY alive_msg:
yacovm (Thu, 15 Nov 2018 22:59:24 GMT):
@h4995974 do you receive it over and over again?
yousaf (Fri, 16 Nov 2018 09:13:35 GMT):
Hi everyone. I have a query, the ports which we assign to our peers in fabric network, why they start from 7050 ? and how much peers can we create? Since we have limited number of ports in our computers like maximum 65535 ??
BellaAdams (Sat, 17 Nov 2018 00:45:59 GMT):
Has joined the channel.
huxiangdong (Tue, 20 Nov 2018 00:31:20 GMT):
Has joined the channel.
Ammu (Thu, 22 Nov 2018 13:28:29 GMT):
node registerUser.js
Store path:/home/ubuntu/.hfc-key-store
Successfully loaded admin from persistence
Failed to register: Error: fabric-ca request register failed with errors [[{"code":20,"message":"Authentication failure"}]]
ivanovv (Fri, 23 Nov 2018 16:15:22 GMT):
Hi,
ivanovv (Fri, 23 Nov 2018 16:26:28 GMT):
Hi,
I have a question, regarding the "requiredPeerCount" parameter on Private Data collection configuration and the corrsponding gossip SendByCriteria() implementation:
Which peers are evaluated to be disseminated to? Is there a preference to disseminate to the Anchor peers of other organisations or all peers (same & different organisations) are equally treated?
Best Regards,
Venko
yacovm (Fri, 23 Nov 2018 16:37:32 GMT):
all peers in the channel
yacovm (Fri, 23 Nov 2018 16:37:38 GMT):
no preference to anchor peers
yacovm (Fri, 23 Nov 2018 16:37:46 GMT):
all peers are equally treated
dave.enyeart (Fri, 23 Nov 2018 16:57:19 GMT):
@yacovm I thought there was a best effort attempt to distribute private data to multiple organizations
yacovm (Fri, 23 Nov 2018 17:01:17 GMT):
nope... that's not how it is
dave.enyeart (Fri, 23 Nov 2018 17:01:46 GMT):
so you might send to N peers within 1 org, and completely miss the other orgs
yacovm (Fri, 23 Nov 2018 17:01:55 GMT):
that's correct
dave.enyeart (Fri, 23 Nov 2018 17:02:08 GMT):
at least the intent was to distribute across orgs best effort, do we need to open a new jira?
yacovm (Fri, 23 Nov 2018 17:02:56 GMT):
do you know a lot of settings where you have a single org that is huge and other smaller satellite orgs with a few peers?
yacovm (Fri, 23 Nov 2018 17:03:33 GMT):
if you put the max peers big enough I think you're good to go
yacovm (Fri, 23 Nov 2018 17:04:16 GMT):
also - when you have a minimum threshold and a maximum threshold
yacovm (Fri, 23 Nov 2018 17:04:38 GMT):
what does it mean to do a best attempt to round-robin across the orgs?
dave.enyeart (Fri, 23 Nov 2018 17:04:41 GMT):
i know a lot of networks have 2-3 peers per org... and it would be better not to double up, rather better to distribute widely. and on the pull side, we could favor peers local to the org
yacovm (Fri, 23 Nov 2018 17:05:39 GMT):
on the pull side we favor the endorsers
dave.enyeart (Fri, 23 Nov 2018 17:05:50 GMT):
best effort would mean distribute to all orgs first before doubling up
dave.enyeart (Fri, 23 Nov 2018 17:06:09 GMT):
yeah, pull side should favor endorsers first, then same org peers, then others
yacovm (Fri, 23 Nov 2018 17:06:25 GMT):
why same org peers? speed?
dave.enyeart (Fri, 23 Nov 2018 17:06:44 GMT):
speed, co-location, trust
yacovm (Fri, 23 Nov 2018 17:06:57 GMT):
you don't need trust, you have the hash
yacovm (Fri, 23 Nov 2018 17:07:08 GMT):
so what is the intent of distributing across orgs ? data availability?
dave.enyeart (Fri, 23 Nov 2018 17:07:13 GMT):
yes
yacovm (Fri, 23 Nov 2018 17:07:28 GMT):
open a JIRA and we'll see what can be done ;)
dave.enyeart (Fri, 23 Nov 2018 17:07:41 GMT):
if you distribute everything to OrgA and then OrgA goes away you're toast
yacovm (Fri, 23 Nov 2018 17:07:55 GMT):
not with reconciliation
dave.enyeart (Fri, 23 Nov 2018 17:08:26 GMT):
we don't want to count on reconciliation, reconciliation is a worst case backstop
yacovm (Fri, 23 Nov 2018 17:08:37 GMT):
ok
yacovm (Fri, 23 Nov 2018 17:08:48 GMT):
so open a JIRA
dave.enyeart (Fri, 23 Nov 2018 17:08:51 GMT):
will do
dave.enyeart (Fri, 23 Nov 2018 17:19:57 GMT):
@yacovm https://jira.hyperledger.org/browse/FAB-12982
yacovm (Fri, 23 Nov 2018 17:22:08 GMT):
@C0rWin ^
yacovm (Fri, 23 Nov 2018 17:24:24 GMT):
@dave.enyeart can this be considered small enough to be backported to v1.4 ?
yacovm (Fri, 23 Nov 2018 17:24:31 GMT):
or do we only backport bug fixes?
dave.enyeart (Fri, 23 Nov 2018 17:29:08 GMT):
We would first get it into master (for v1.4). If it passes all tests well including system tests, and if the code appears low risk, we can consider backport to v1.3 at that time.
dave.enyeart (Fri, 23 Nov 2018 17:29:40 GMT):
That being said, v1.4 is our target long term support release and we'd encourage people to move to v1.4
dave.enyeart (Fri, 23 Nov 2018 17:30:19 GMT):
Backporting functions is a slippery slope, need to be very judicious about it
dave.enyeart (Fri, 23 Nov 2018 17:30:19 GMT):
Backporting enhancements is a slippery slope, need to be very judicious about it
yacovm (Fri, 23 Nov 2018 17:41:50 GMT):
@dave.enyeart what's the time window for v1.4 development? isn't that end of month?
ivanovv (Mon, 26 Nov 2018 08:56:19 GMT):
@yacovm (comment on post @6:06 PM)
I have a simple setup: 1 host, 1 docker network, 3 orgs x 1 peer.
OrgA writes to private collections onlyA & onlyB. Setting is for both "requiredPeerCount=0". The data is Not written to the ponlyB database within an hour+ (the hash ends up in the honlyB database almost immediately).
After setting up "requiredPeerCount=1" the data was available for OrgB within seconds (as expected).
These are just observations for those 2 use cases. I definitely have to play more on the case, but it seems (for now to me, and it could be that I'm wrong) that being able to push targeted during the endorsement phase helps.
Thanks,
Venko
C0rWin (Mon, 26 Nov 2018 09:03:46 GMT):
@ivanovv setting `requiredPeerCount=0` has semantic of not distributing private data at endorsement time
ivanovv (Mon, 26 Nov 2018 09:08:02 GMT):
@C0rWin Yes, I expected that... But it "never" occurred. At least not within 1+ hour.
ivanovv (Mon, 26 Nov 2018 09:08:02 GMT):
OrgA writes to private collections onlyA & onlyB. Setting is for both "requiredPeerCount=0". The data is Not written to the ponlyB database within an hour+""
ivanovv (Mon, 26 Nov 2018 09:08:02 GMT):
Yes
C0rWin (Mon, 26 Nov 2018 09:08:28 GMT):
not following you
C0rWin (Mon, 26 Nov 2018 09:09:18 GMT):
if it's expected it shall never occur even after 100+ hours
C0rWin (Mon, 26 Nov 2018 09:10:05 GMT):
oh, I see, you are saying that peer of ponlyB, commits block and doesn't pull the data?
ivanovv (Mon, 26 Nov 2018 09:10:51 GMT):
and with requiredPeersCout=1 we will have feedback
C0rWin (Mon, 26 Nov 2018 09:12:32 GMT):
@ivanovv on what version are you testing it?
ivanovv (Mon, 26 Nov 2018 09:12:36 GMT):
1.3
ivanovv (Mon, 26 Nov 2018 09:14:50 GMT):
@C0rWin Could you please confirm that with "requiredPeerCount=0"
"if it's expected it shall (dissemination) never occur even after 100+ hours"
C0rWin (Mon, 26 Nov 2018 09:15:10 GMT):
let me clarify
C0rWin (Mon, 26 Nov 2018 09:15:23 GMT):
there are two aspects of data dissemination
C0rWin (Mon, 26 Nov 2018 09:16:02 GMT):
while tx endorsed peer should try to push private data based on `requiredPeerCount` number of peers
C0rWin (Mon, 26 Nov 2018 09:17:10 GMT):
so if this is set to 0 peer will never tries to push anything
C0rWin (Mon, 26 Nov 2018 09:17:19 GMT):
however
C0rWin (Mon, 26 Nov 2018 09:18:09 GMT):
there is a second part, where peer while getting next block to commit extract info about missing private data pieces and tries to pull them
C0rWin (Mon, 26 Nov 2018 09:18:43 GMT):
what I meant to say is that with `requiredPeerCount=0` data will be never pushed, even after 100 hours :)
C0rWin (Mon, 26 Nov 2018 09:18:58 GMT):
but definitely peer should be able to pull it during commit time
ivanovv (Mon, 26 Nov 2018 09:20:25 GMT):
And in my case the second phase did not happen within an hour+
ivanovv (Mon, 26 Nov 2018 09:21:31 GMT):
And according one video there is a timeout of about an hour after which the second peer will dtop trying to retreave the data and this peer will not be able to deliver the private data anymore.
ivanovv (Mon, 26 Nov 2018 09:21:31 GMT):
And according to one video there is a timeout of about an hour after which the second peer will dtop trying to retreave the data and this peer will not be able to deliver the private data anymore.
ivanovv (Mon, 26 Nov 2018 09:21:31 GMT):
And according to one video there is a timeout of about an hour after which the second peer will stop trying to retrieve the data and this peer will not be able to deliver the private data anymore.
ivanovv (Mon, 26 Nov 2018 09:21:31 GMT):
And according to one video there is a timeout of about an hour after which the second peer will stop trying to retrieve the data and this peer will not be able to deliver the private data to the apps anymore.
C0rWin (Mon, 26 Nov 2018 09:26:33 GMT):
can you run your peers in debug mode?
C0rWin (Mon, 26 Nov 2018 09:26:38 GMT):
and collect logs?
C0rWin (Mon, 26 Nov 2018 09:26:54 GMT):
I'd be glad to take a look on this closer
ivanovv (Mon, 26 Nov 2018 09:32:13 GMT):
@C0rWin Yes, I will do. Is there something in the logs that deserves to pay more attention at? I believe I can stop logging about a minute after the transaction gets committed (everything is local 3 orgs x 1 peer & 2 cpu's). requiredPeerCount will be 0. OK?
ivanovv (Mon, 26 Nov 2018 09:58:04 GMT):
@C0rWin
Hi, I believe that even with requiredPeerCount=0 the peer will try to disseminate messages to (up to the maxPeerCount) other peers.
It will just not throw an exception.
My conclusion is based on the code in:
https://github.com/hyperledger/fabric/blob/c45fde94b72a0f2fad7db37dff54d08ed4b280a8/gossip/gossip/gossip_impl.go
line: 645 & 654.
Best regards,
Venko Ivanov
PS. the same was implicitly suggested by Dave & yakovm.
ivanovv (Mon, 26 Nov 2018 09:58:04 GMT):
@dave.enyeart
Hi, I believe that even with requiredPeerCount=0 the peer will try to disseminate messages to (up to the maxPeerCount) other peers.
It will just not throw an exception.
My conclusion is based on the code in:
https://github.com/hyperledger/fabric/blob/c45fde94b72a0f2fad7db37dff54d08ed4b280a8/gossip/gossip/gossip_impl.go
line: 645 & 654.
Best regards,
Venko Ivanov
PS. the same was implicitly suggested by Dave & yakovm.
dave.enyeart (Mon, 26 Nov 2018 11:52:18 GMT):
@ivanovv that's right. what was your maxPeerCount? Still need your debug for both A and B too understand why thee private data wasn't pulled at commit time.
dave.enyeart (Mon, 26 Nov 2018 11:52:18 GMT):
@ivanovv that's right. what was your maxPeerCount? Still need your debug for both A and B too understand why the private data wasn't pulled at commit time.
C0rWin (Mon, 26 Nov 2018 12:09:32 GMT):
@ivanovv can you please post here you collection config?
ivanovv (Mon, 26 Nov 2018 13:30:51 GMT):
@C0rWin Most probably this was the problem.
Looking in the code for my previous post I saw that with maxPeerCount = 0 it will not try to disseminate at endorsement time at all...
I've fixed this (based on info I've got from David on the fabric-ledger thread) last Friday and it worked...
The reason I expected it to work was based on previous tests (AFAI remember) when OrgA was writing to collectionOrgA&OrgB and everything worked with both params set to 0.
Thanks for the help!
Best regards,
Venko
ivanovv (Mon, 26 Nov 2018 13:30:51 GMT):
@dave.enyeart Most probably this was the problem.
Looking in the code for my previous post I saw that with maxPeerCount = 0 it will not try to disseminate at endorsement time at all...
I've fixed this (based on info I've got from David on the fabric-ledger thread) last Friday and it worked...
The reason I expected it to work was based on previous tests (AFAI remember) when OrgA was writing to collectionOrgA&OrgB and everything worked with both params set to 0.
Thanks for the help!
Best regards,
Venko
ivanovv (Mon, 26 Nov 2018 13:30:51 GMT):
@dave.enyeart Most probably this was the problem.
Looking in the code for my previous post I saw that with maxPeerCount = 0 it will not try to disseminate at endorsement time at all...
I've fixed this (based on info I've got from you on the fabric-ledger thread) last Friday and it worked...
The reason I expected it to work was based on previous tests (AFAI remember) when OrgA was writing to collectionOrgA&OrgB and everything worked with both params set to 0.
Thanks for the help!
Best regards,
Venko
joaquimpedrooliveira (Tue, 27 Nov 2018 13:40:06 GMT):
Has left the channel.
me020523 (Wed, 28 Nov 2018 03:14:25 GMT):
Has joined the channel.
migrenaa (Fri, 30 Nov 2018 08:23:14 GMT):
Hello. I am running a hyperledger fabric network with 4 kafkas, for my dev env, they are running on a single VM. Yesterday my VM ran out of memory and 3 of the kafkas died. The last one threw an error that a leader election failed. If a leader kafka dies, one of the followers should become a leader immediately. So even with one kafka the network shouldn't die. I tried to reproduce the issue, I am killing kafka containers, but now everything is working fine. Do you have an idea what might cause the problem?
migrenaa (Fri, 30 Nov 2018 08:23:14 GMT):
Hello. I am running a hyperledger fabric network with 4 kafkas, 1 peer, 2 orderers, for my dev env, they are running on a single VM. Yesterday my VM ran out of memory and 3 of the kafkas died. The last one threw an error that a leader election failed. If a leader kafka dies, one of the followers should become a leader immediately. So even with one kafka the network shouldn't die. I tried to reproduce the issue, I am killing kafka containers, but now everything is working fine. Do you have an idea what might cause the problem?
migrenaa (Fri, 30 Nov 2018 08:23:14 GMT):
Hello. I am running a hyperledger fabric network with 4 kafkas, 1 peer, 2 orderers, for my dev env, they are running on a single VM. Yesterday the VM ran out of memory and 3 of the kafkas died. The last one threw an error that a leader election failed. If a leader kafka dies, one of the followers should become a leader immediately. So even with one kafka the network should work fine I tried to reproduce the issue, I am killing kafka containers, but now everything is working fine. Do you have an idea what might cause the problem?
yacovm (Fri, 30 Nov 2018 09:32:40 GMT):
@migrenaa this isn't related to fabric gossip
yacovm (Fri, 30 Nov 2018 09:32:47 GMT):
please post it in #fabric-orderer
migrenaa (Fri, 30 Nov 2018 10:00:58 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=ibnHRps2zkHd58Qkz) @yacovm ok thanks :)
dongming (Sat, 01 Dec 2018 16:29:05 GMT):
Has joined the channel.
dongming (Sat, 01 Dec 2018 16:30:24 GMT):
@yacovm I am trying to update the anchor peer in the channel, and I see the following warning message in the peer log
```
2018-12-01 16:26:06.353 UTC [gossip/gossip] learnAnchorPeers -> INFO 070 Learning about the configured anchor peers of PeerOrg4 for channel testorgschannel1 : [{peer0.org4.example.com 7067}]
2018-12-01 16:26:06.353 UTC [gossip/service] updateEndpoints -> WARN 071 Failed to update ordering service endpoints, due to Channel with testorgschannel1 id was not found
2018-12-01 16:26:06.357 UTC [committer/txvalidator] Validate -> INFO 072 [testorgschannel1] Validated block [5] in 122ms
2018-12-01 16:26:06.373 UTC [kvledger] CommitWithPvtData -> INFO 073 [testorgschannel1] Committed block [5] with 1 transaction(s) in 10ms (state_validation=1ms block_commit=6ms state_commit=1ms)
2018-12-01 16:26:06.394 UTC [gossip/comm] func1 -> WARN 074 peer0.org3.example.com:7065, PKIid:[72 232 27 3 230 161 244 55 230 28 218 144 10 30 129 109 90 176 107 8 194 47 55 77 221 198 180 215 207 169 228 195] isn't responsive: EOF
2018-12-01 16:26:06.394 UTC [gossip/discovery] expireDeadMembers -> WARN 075 Entering [[72 232 27 3 230 161 244 55 230 28 218 144 10 30 129 109 90 176 107 8 194 47 55 77 221 198 180 215 207 169 228 195]]
2018-12-01 16:26:06.394 UTC [gossip/discovery] expireDeadMembers -> WARN 076 Closing connection to Endpoint: peer0.org3.example.com:7065, InternalEndpoint: , PKI-ID: [72 232 27 3 230 161 244 55 230 28 218 144 10 30 129 109 90 176 107 8 194 47 55 77 221 198 180 215 207 169 228 195], Metadata: []
2018-12-01 16:26:06.394 UTC [gossip/discovery] expireDeadMembers -> WARN 077 Exiting
```
dongming (Sat, 01 Dec 2018 16:31:00 GMT):
Does this mean that the peer cannot communicate with other anchor peers on the channel
yacovm (Sat, 01 Dec 2018 16:33:08 GMT):
no
arjitkhullar (Wed, 05 Dec 2018 00:02:19 GMT):
Has joined the channel.
Pradeep_Pentakota (Wed, 05 Dec 2018 01:41:16 GMT):
Has joined the channel.
edwardlee (Thu, 13 Dec 2018 14:56:10 GMT):
Has joined the channel.
aviralwal (Mon, 17 Dec 2018 11:49:04 GMT):
Has joined the channel.
merq (Sat, 22 Dec 2018 03:45:03 GMT):
Has joined the channel.
deelthor (Mon, 07 Jan 2019 10:04:29 GMT):
Has joined the channel.
deelthor (Mon, 07 Jan 2019 10:04:36 GMT):
Hey guys! In our current setup we have 2 peers and 2 orderers. We have a specific transaction which is executed every 15 min. So every 15 min I see something like this in the Peer Logs: `2018-10-11 01:15:29.224 UTC [gossip/privdata] StoreBlock -> INFO 719c Received block [33327]`
`2018-10-11 01:15:29.758 UTC [kvledger] CommitWithPvtData -> INFO 719d Channel [bccchannel]: Committed block [33327] with 1 transaction(s)` . After sometime though the logs wont show up anymore and nothing is written to the ledger. After a restart of the peer, the logs show up again. My assumption is that the connection between Orderer and Peer gets lost/was closed. Does anyone have an idea why this could happen? Any help is very much appreciated!
C0rWin (Mon, 07 Jan 2019 10:46:18 GMT):
> nothing is written to the ledger
@deelthor how do you check this? do you see any errors on the log? are you subscribed to events? have you checked tx status to see it's valid?
C0rWin (Mon, 07 Jan 2019 10:46:18 GMT):
> nothing is written to the ledger
@deelthor how do you check this? do you see any errors on the log? are you subscribed to events? have you checked tx status to see it's valid?
C0rWin (Mon, 07 Jan 2019 10:46:18 GMT):
> nothing is written to the ledger
how do you check this? do you see any errors on the log? are you subscribed to events? have you checked tx status to see it's valid? @deelthor
deelthor (Mon, 07 Jan 2019 11:38:41 GMT):
I cannot query the data which should be created by the transaction. After a restart of the peers I can see the data appearing.
deelthor (Mon, 07 Jan 2019 11:38:41 GMT):
I cannot query the data which should be created by the transaction. After a restart of the peers I can see the data appearing. @C0rWin
deelthor (Tue, 08 Jan 2019 18:16:12 GMT):
Does anyone have any idea why / when the previously described behaviour might occur?
C0rWin (Tue, 08 Jan 2019 22:57:14 GMT):
@deelthor do you see anything else in the logs? it block is committed, hence you should be able to see the data. are you sure you client doesn't cache anything?
spacemandev (Fri, 11 Jan 2019 12:11:32 GMT):
Has joined the channel.
robertbrown (Fri, 11 Jan 2019 19:50:41 GMT):
Has joined the channel.
ycarmel (Tue, 22 Jan 2019 08:44:54 GMT):
Has joined the channel.
Jamie (Tue, 22 Jan 2019 17:09:25 GMT):
Has joined the channel.
incarose (Wed, 23 Jan 2019 00:23:22 GMT):
Has joined the channel.
allanlee2019 (Thu, 24 Jan 2019 17:14:28 GMT):
Has joined the channel.
binhn (Sat, 26 Jan 2019 16:50:00 GMT):
Has left the channel.
lip-inagora (Mon, 28 Jan 2019 00:22:06 GMT):
Has joined the channel.
samirbalabantaray (Tue, 29 Jan 2019 09:25:20 GMT):
Has joined the channel.
JayJong (Fri, 08 Feb 2019 10:55:05 GMT):
```2019-02-08 10:54:44.112 UTC [gossip/comm] GossipStream -> ERRO e9342 Authentication failed: failed classifying identity: Unable to extract msp.Identity from peer Identity: Peer Identity``` Any1 faced this issue before?
JayJong (Fri, 08 Feb 2019 10:55:05 GMT):
```2019-02-08 10:54:44.112 UTC [gossip/comm] GossipStream -> ERRO e9342 Authentication failed: failed classifying identity: Unable to extract msp.Identity from peer Identity: Peer Identity[] cannot be validated. No MSP found able to do that.``` Any1 faced this issue before?
yacovm (Fri, 08 Feb 2019 12:01:52 GMT):
you haven't added the anchor peer to the channel @JayJong
JayJong (Sat, 09 Feb 2019 21:30:47 GMT):
@yacovm thanks for the pointer! I have another qn, im using fabric v1.3, after setting the network with 15 peers and installed chaincode, my peer logs kept increasing to the point where i do not have enough disk space because of gossip/gossip and gossip/comm DEBUG logs. Is this normal? Is it possible that i dun receive the debug logs and just listen to ERROR logs?
yacovm (Sat, 09 Feb 2019 21:32:02 GMT):
yeah it's possible
yacovm (Sat, 09 Feb 2019 21:32:28 GMT):
set the environment variable `CORE_LOGGING_GOSSIP=WARNING`
JayJong (Sun, 10 Feb 2019 15:39:08 GMT):
I want to have gossip of WARNING level and the rest at DEBUG level. I defined CORE_LOGGING_LEVEL=DEBUG and CORE_LOGGING_GOSSIP=WARNING in my docker compose but it seems that gossip was still at debug level. Any way i could get what i want?
yacovm (Sun, 10 Feb 2019 15:54:51 GMT):
@wlahti ^
knagware9 (Mon, 11 Feb 2019 05:34:50 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=3Zyktty9yuwCjrRjJ) @JayJong The logging levels of the peer and orderer commands are controlled by a logging specification, which is set via the FABRIC_LOGGING_SPEC environment variable.
knagware9 (Mon, 11 Feb 2019 05:35:57 GMT):
may be that is the reason its still in debug mode , if you try with FABRIC_LOGGING_SPEC = WARNING , may be it works
JayJong (Mon, 11 Feb 2019 06:27:18 GMT):
@knagware9 thanks for the reply, im aware that FABRIC_LOGGING_SPEC has replaced CORE_LOGGING_LEVEL in v1.4. My current workabout for my situation is that im writing CORE_LOGGING_LEVEL=debug:gossip/gossip,gossip/comm,gossip/discovery,gossip/pull,gossip/election=warning
knagware9 (Mon, 11 Feb 2019 06:40:45 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=vYmgmFXNwwe5sxLvi) @JayJong Okay
wlahti (Mon, 11 Feb 2019 12:46:48 GMT):
@JayJong I'm assuming you're using a version of Fabric pre-1.4? And yes, in the later versions before 1.4, the environment variable overrides (e.g. CORE_LOGGING_GOSSIP, CORE_LOGGING_MSP) aren't really functional. Some concurrency issues with the logging package we used required some changes that "broke" those overrides. Glad you found the spec functionality that's always been available via CORE_LOGGING_LEVEL.
JayJong (Mon, 11 Feb 2019 17:01:37 GMT):
@wlahti yes, im using v1.3 currently but im upgrading to v1.4 soon. Do i still have to do `FABRIC_LOGGING_SPEC=debug:gossip/gossip,gossip/comm,gossip/discovery,gossip/pull,gossip/election=warning` for v1.4 or is there a cleaner way?
wlahti (Mon, 11 Feb 2019 17:03:02 GMT):
for v1.4, you should be able to use `FABRIC_LOGGING_SPEC=debug:gossip=warning` to set any logger that begins with `gossip` to warning
JayJong (Tue, 12 Feb 2019 02:34:51 GMT):
@wlahti Ok, thanks for the help!
SatoshiNishishita (Wed, 13 Feb 2019 01:28:06 GMT):
Has joined the channel.
danacr (Thu, 14 Feb 2019 20:05:05 GMT):
Has left the channel.
ping40 (Fri, 22 Feb 2019 04:13:38 GMT):
Has joined the channel.
cjml1982 (Thu, 07 Mar 2019 07:32:53 GMT):
Has joined the channel.
jtrayfield (Mon, 11 Mar 2019 19:11:05 GMT):
Has joined the channel.
KyunghoKim (Tue, 12 Mar 2019 03:08:59 GMT):
Has joined the channel.
MHBauer (Tue, 12 Mar 2019 18:49:17 GMT):
Has joined the channel.
jtrayfield (Tue, 12 Mar 2019 21:51:15 GMT):
Hi, I have multiple organizations with one peer per org. when I instantiate chaincode, Init() is only called in the chaincode for the peer I instantiate on. it's never called on the other peers.
how is the data that's set in Init() supposed to get to the other peers? is it supposed to be propagated by gossip?
yacovm (Tue, 12 Mar 2019 22:14:37 GMT):
the init transactions gets into the next block, @jtrayfield
yacovm (Tue, 12 Mar 2019 22:14:45 GMT):
and it gets either from orderer or via gossip
yanli133 (Mon, 25 Mar 2019 06:56:28 GMT):
Has joined the channel.
krishnatejap (Mon, 25 Mar 2019 11:47:20 GMT):
Has joined the channel.
krishnatejap (Mon, 25 Mar 2019 12:27:06 GMT):
Hi All,
We are intending to run a fabric network with 26 organizations comprising of:
- one orderer ("solo" based consensus)
- two peers per organizations each having a couchdb database. Out of two, one peer is designated
as the Anchor peer and are setting "CORE_PEER_GOSSIP_USELEADERELECTION=true" for electing the
leader dynamically.
On the whole, we are spinning 105 containers (52 peers + 52 couchdb + one orderer).
We based our network configuration on the "BYFN" example. In addition to this,
we are setting the values of "CORE_PEER_TLS_CLIENTAUTHREQUIRED" and "ORDERER_GENERAL_TLS_CLIENTAUTHREQUIRED"
to "true" - also set the value of "ORDERER_GENERAL_TLS_CLIENTROOTCAS" to contain the list of client CA's.
We were able to successfully spin off a network with 14 organizations but when tried to extend it for 12 more
organizations, we are facing multiple issues - by extending, I meant we are trying to create the peers for
26 organizations at one go. We are running the network on a single machine.
We were able to create the channel and join the peers to the channel. But when doing an "Anchor" peer update,
the network stops when updating the "Anchor" peer for the 19th organization with the following error: "Context deadline exceeded."
We added a sleep of 100 seconds between each update and set the following environment variables.
- CORE_PEER_GOSSIP_ALIVETIMEINTERVAL=100s
- CORE_PEER_GOSSIP_ALIVEEXPIRATIONTIMEOUT=100s
- CORE_PEER_GOSSIP_RECONNECTINTERVAL=100s
- CORE_PEER_GOSSIP_DIALTIMEOUT=1000s
- CORE_PEER_GOSSIP_RESPONSEWAITTIME=1000s
- CORE_PEER_DISCOVERY_PERIOD=300s
- CORE_PEER_DISCOVERY_TOUCHPERIOD=30s
- CORE_LEDGER_STATE_COUCHDBCONFIG_REQUESTTIMEOUT=300s
- CORE_PEER_KEEPALIVE=300s
- CORE_PEER_KEEPALIVE_CLIENT_INTERVAL=300s
We aren't sure if they are required, but assuming that it would, we tried but failed.
This time, the anchor peer update was successful, despite the following errors/warnings:
2019-03-25 11:04:35.518 UTC [gossip/comm] sendToEndpoint -> WARN a8c Failed obtaining connection for peer0.organization17.com:7051, PKIid:[213 58 75 254 200 30 208 175 202 1 180 210 202 31 30 144 176 199 65 190 121 87 70 35 133 236 10 32 167 113 116 112] reason: context deadline exceeded
2019-03-25 11:07:36.109 UTC [gossip/discovery] func1 -> WARN b4c Could not connect to {peer0.organization6.com:7051 [] [] peer0.organization6.com:7051
yacovm (Mon, 25 Mar 2019 15:06:58 GMT):
are you running in docker?
krishnatejap (Mon, 25 Mar 2019 16:47:28 GMT):
Hi @yacovm , we are not running in docker.
krishnatejap (Mon, 25 Mar 2019 16:48:27 GMT):
@yacovm , We tried setting up the system using the Fabric versions V1.3 and V1.4 but were unable to do so.
krishnatejap (Mon, 25 Mar 2019 17:20:04 GMT):
@yacovm , each peer and the orderer is run as a container.
krishnatejap (Mon, 25 Mar 2019 17:20:14 GMT):
Please excuse me for the confusion
krishnatejap (Wed, 27 Mar 2019 15:04:07 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-gossip?msg=4yCGxsNv6rojxeRWX) Hi Team, any update on the issue please?
yacovm (Wed, 27 Mar 2019 15:05:58 GMT):
try to see why it doesn't connect
yacovm (Wed, 27 Mar 2019 15:06:04 GMT):
run `tcpdump -w out.log`
yacovm (Wed, 27 Mar 2019 15:06:06 GMT):
and open it with wireshark
yacovm (Wed, 27 Mar 2019 15:06:17 GMT):
and see if this is a network problem or a TLS issue
plato (Mon, 01 Apr 2019 22:02:10 GMT):
Has joined the channel.
plato (Mon, 01 Apr 2019 22:05:05 GMT):
@yacovm I have 5 orgs on my network and I can invoke the chaincode function and everything works fine even private collection works well on my network but I'm facing with this WARN cfd Message GossipMessage: tag:CHAN_OR_ORG state_info:
plato (Mon, 01 Apr 2019 22:05:05 GMT):
@yacovm I have 5 orgs on my network and I can invoke the chaincode function and everything works fine even private collection works well on my network but I'm facing with this WARN cfd Message GossipMessage: tag:CHAN_OR_ORG state_info:
plato (Mon, 01 Apr 2019 22:06:36 GMT):
gossip.gossip] handleMessage -> WARN
plato (Mon, 01 Apr 2019 22:06:36 GMT):
gossip.gossip] handleMessage -> WARN
yacovm (Tue, 02 Apr 2019 12:32:41 GMT):
does it repeat?
xixuejia (Tue, 02 Apr 2019 13:11:47 GMT):
Hello @yacovm , when calling service discovery from a peer, I encountered an issue that the peer only returned a sub set of endorsers from time to time. How long should I wait before a peer can get a full list of endorsers list for a specific chaincode?
yacovm (Tue, 02 Apr 2019 13:19:09 GMT):
@xixuejia hmm you can see why it doesn't select all of them
yacovm (Tue, 02 Apr 2019 13:19:15 GMT):
for example try to do a peer membership query
yacovm (Tue, 02 Apr 2019 13:19:20 GMT):
and see if it has them in the view
xixuejia (Tue, 02 Apr 2019 13:27:53 GMT):
if I repeat doing service discovery, say I have 2 orgs with 2 peers per org, the peer will might return 1, 2, 3, 4 peers
xixuejia (Tue, 02 Apr 2019 13:28:03 GMT):
that seems unpredicatable
yacovm (Tue, 02 Apr 2019 13:29:29 GMT):
when you do a peer query?
yacovm (Tue, 02 Apr 2019 13:29:32 GMT):
@xixuejia
xixuejia (Tue, 02 Apr 2019 13:30:08 GMT):
yes
yacovm (Tue, 02 Apr 2019 13:41:11 GMT):
and the peer query returns to you not all peers?
xixuejia (Tue, 02 Apr 2019 13:48:25 GMT):
correct
xixuejia (Tue, 02 Apr 2019 13:48:52 GMT):
I did encounter this in v1.3 and v1.4
yacovm (Tue, 02 Apr 2019 13:49:34 GMT):
hmm this is most likely not a bug in discovery, but actually the peers themselves have an unstable membership view
yacovm (Tue, 02 Apr 2019 13:49:52 GMT):
check the logs to see if the peers complain about some errors or somethign
yacovm (Tue, 02 Apr 2019 13:50:08 GMT):
and check if the peers communicate with one another
yacovm (Tue, 02 Apr 2019 13:50:53 GMT):
if it is a bug somewhere, i would guess it might be a bug in gossip that it sometimes doesn't return an external endpoint
yacovm (Tue, 02 Apr 2019 13:51:19 GMT):
but i don't think it can be a bug in discovery... the code path for membership query is very simple
yacovm (Tue, 02 Apr 2019 13:51:26 GMT):
so check for the logs first please :)
xixuejia (Tue, 02 Apr 2019 13:52:22 GMT):
I can see logs like ```2019-04-02 13:32:36.132 UTC [nodeCmd] func9 -> INFO 01d Starting profiling server with listenAddress = 0.0.0.0:6060
2019-04-02 13:32:37.129 UTC [gossip/comm] func1 -> WARN 01e peer1.org1.example.com:7051, PKIid:[109 3 56 61 77 38 219 195 76 245 21 241 45 192 238 190 205 104 103 68 115 35 52 204 70 155 222 253 23 78 73 47] isn't responsive: rpc error: code = Unavailable desc = transport is closing
2019-04-02 13:32:37.129 UTC [gossip/discovery] expireDeadMembers -> WARN 01f Entering [[109 3 56 61 77 38 219 195 76 245 21 241 45 192 238 190 205 104 103 68 115 35 52 204 70 155 222 253 23 78 73 47]]
2019-04-02 13:32:37.129 UTC [gossip/discovery] expireDeadMembers -> WARN 020 Closing connection to Endpoint: peer1.org1.example.com:7051, InternalEndpoint: peer1.org1.example.com:7051, PKI-ID: [109 3 56 61 77 38 219 195 76 245 21 241 45 192 238 190 205 104 103 68 115 35 52 204 70 155 222 253 23 78 73 47], Metadata: []
2019-04-02 13:32:37.129 UTC [gossip/discovery] expireDeadMembers -> WARN 021 Exiting```
yacovm (Tue, 02 Apr 2019 13:52:37 GMT):
aha
yacovm (Tue, 02 Apr 2019 13:52:41 GMT):
ok so it's because of it
xixuejia (Tue, 02 Apr 2019 13:52:58 GMT):
I don't know why it expire dead members. all these peers run in a single machine
yacovm (Tue, 02 Apr 2019 13:53:12 GMT):
how many peers do you have?
xixuejia (Tue, 02 Apr 2019 13:53:23 GMT):
2*2=4 peers
yacovm (Tue, 02 Apr 2019 13:53:31 GMT):
how many channels?
xixuejia (Tue, 02 Apr 2019 13:53:35 GMT):
10
yacovm (Tue, 02 Apr 2019 13:59:22 GMT):
oh i see
yacovm (Tue, 02 Apr 2019 13:59:27 GMT):
so, you need more CPUs :(
yacovm (Tue, 02 Apr 2019 13:59:40 GMT):
the message verification is too costly and i never had the time to optimize it
yacovm (Tue, 02 Apr 2019 14:00:07 GMT):
check `top`
yacovm (Tue, 02 Apr 2019 14:00:12 GMT):
see if the CPU is on 100%
xixuejia (Tue, 02 Apr 2019 14:02:35 GMT):
:ok_hand: haha. let me check. Thanks for your help @yacovm
xixuejia (Tue, 02 Apr 2019 14:04:07 GMT):
it's not 100%, average to ~50%
yacovm (Tue, 02 Apr 2019 14:06:17 GMT):
hmmm odd.
yacovm (Tue, 02 Apr 2019 14:06:32 GMT):
so you don't need more CPUs then i think
yacovm (Tue, 02 Apr 2019 14:06:49 GMT):
but seems like the peer only started
yacovm (Tue, 02 Apr 2019 14:06:54 GMT):
it takes some time for things to stabilize
yacovm (Tue, 02 Apr 2019 14:07:01 GMT):
if you want like 30 seconds
yacovm (Tue, 02 Apr 2019 14:07:04 GMT):
how is that then?
xixuejia (Tue, 02 Apr 2019 14:16:49 GMT):
sure, I will retry it later. Thanks @yacovm
xixuejia (Tue, 09 Apr 2019 05:26:35 GMT):
Hi @yacovm with the suggestions you provided last week, I found that the cpu usage was ~100% when I do cc instantiate, and I did service discovery immediately after cc instantiate, I think that caused the issue I encountered. I gave it a 5s sleep after cc instantiate and I no longer encounter that problem. Thanks for your help!
yacovm (Tue, 09 Apr 2019 07:57:24 GMT):
@xixuejia cool
yacovm (Tue, 09 Apr 2019 07:57:28 GMT):
glad i could help
zacpl (Wed, 10 Apr 2019 11:28:48 GMT):
I have a quick question about private data propagation:
Starting in v 1.4 peers from organizations that are added to a private data collection definition automatically fetch private data committed to the collection before they joined.
My question is whether they receive historical write-set information (so that this information can be used to audit past transactions) - or whether they just receive contemporary data?
My question is related to this: https://stackoverflow.com/questions/51960671/history-for-private-data-in-hyperledger-fabric
Help clarifying these points appreciated.
dave.enyeart (Wed, 10 Apr 2019 20:29:26 GMT):
@zacpl in v1.4... peers from newly added orgs will receive the historical private write-sets for collections that they join, and their private state database will be updated if the private data represents the latest keys/values. The piece that is not there is APIs to query this information: https://jira.hyperledger.org/browse/FAB-5094
zacpl (Thu, 11 Apr 2019 11:06:55 GMT):
Thanks @dave.enyeart !
Somehow missed that on jira: only spotted FAB-5093...
will stay posted
mistogans (Wed, 24 Apr 2019 06:59:05 GMT):
Has joined the channel.
mistogans (Thu, 25 Apr 2019 01:57:11 GMT):
Hi, we are seeing problems with the gossip membership list on every peer in our Fabric network. In summary, the membership list is very unstable and peers constantly drop out of each other's list every few seconds. Below is a snippet of our logs.
```2019-04-25 01:51:55.608 UTC [gossip.discovery] expireDeadMembers -> WARN d202 Exiting
2019-04-25 01:51:59.140 UTC [gossip.channel] reportMembershipChanges -> INFO d203 Membership view has changed. peers went offline: [[peer0.org11.example.com:7051 ]] , current view: [[peer0.org10.example.com:7051 ] [peer0.org13.example.com:7051 ] [peer0.org19.example.com:7051 ] [peer0.org6.example.com:7051 ] [peer0.org14.example.com:7051 ] [peer0.org15.example.com:7051 ] [peer0.org4.example.com:7051 ] [peer0.org7.example.com:7051 ] [peer0.org23.example.com:7051 ] [peer0.org18.example.com:7051 ] [peer0.org21.example.com:7051 ] [peer0.org16.example.com:7051 ] [peer0.org5.example.com:7051 ] [peer0.org9.example.com:7051 ] [peer0.org22.example.com:7051 ] [peer0.org20.example.com:7051 ] [peer0.org12.example.com:7051 ] [peer0.org8.example.com:7051 ] [peer0.org17.example.com:7051 ] [peer0.org1.example.com:7051 ]]
2019-04-25 01:52:05.712 UTC [comm.grpc.server] 1 -> INFO d204 unary call completed {"grpc.start_time": "2019-04-25T01:52:05.712Z", "grpc.service": "gossip.Gossip", "grpc.method": "Ping", "grpc.request_deadline": "2019-04-25T01:52:07.712Z", "grpc.peer_address": "10.0.9.41:44594", "grpc.peer_subject": "CN=peer0.org22.example.com,L=San Francisco,ST=California,C=US", "grpc.code": "OK", "grpc.call_duration": "157.975µs"}
2019-04-25 01:52:09.140 UTC [gossip.channel] reportMembershipChanges -> INFO d205 Membership view has changed. peers went online: [[peer0.org11.example.com:7051 ]] , current view: [[peer0.org15.example.com:7051 ] [peer0.org6.example.com:7051 ] [peer0.org14.example.com:7051 ] [peer0.org23.example.com:7051 ] [peer0.org18.example.com:7051 ] [peer0.org21.example.com:7051 ] [peer0.org16.example.com:7051 ] [peer0.org5.example.com:7051 ] [peer0.org9.example.com:7051 ] [peer0.org4.example.com:7051 ] [peer0.org7.example.com:7051 ] [peer0.org20.example.com:7051 ] [peer0.org22.example.com:7051 ] [peer0.org17.example.com:7051 ] [peer0.org1.example.com:7051 ] [peer0.org12.example.com:7051 ] [peer0.org8.example.com:7051 ] [peer0.org11.example.com:7051 ] [peer0.org19.example.com:7051 ] [peer0.org10.example.com:7051 ] [peer0.org13.example.com:7051 ]]
2019-04-25 01:52:09.353 UTC [comm.grpc.server] 1 -> INFO d206 streaming call completed {"grpc.start_time": "2019-04-25T01:49:29.299Z", "grpc.service": "gossip.Gossip", "grpc.method": "GossipStream", "grpc.peer_address": "10.0.9.3:39924", "grpc.peer_subject": "CN=peer0.org13.example.com,L=San Francisco,ST=California,C=US", "error": "rpc error: code = Canceled desc = context canceled", "grpc.code": "Canceled", "grpc.call_duration": "2m40.05450849s"}
2019-04-25 01:52:13.110 UTC [gossip.discovery] getDeadMembers -> WARN d207 Haven't heard from [228 6 93 182 126 13 90 123 11 24 161 26 85 17 97 232 130 132 194 172 229 28 198 59 93 41 219 118 182 70 42 5] for 25.629033151s
2019-04-25 01:52:13.113 UTC [gossip.discovery] expireDeadMembers -> WARN d208 Entering [e4065db67e0d5a7b0b18a11a551161e88284c2ace51cc63b5d29db76b6462a05]
2019-04-25 01:52:13.113 UTC [gossip.discovery] expireDeadMembers -> WARN d209 Closing connection to Endpoint: peer0.org18.example.com:7051, InternalEndpoint: , PKI-ID: e4065db67e0d5a7b0b18a11a551161e88284c2ace51cc63b5d29db76b6462a05, Metadata:
2019-04-25 01:52:13.114 UTC [gossip.discovery] expireDeadMembers -> WARN d20a Exiting
2019-04-25 01:52:14.140 UTC [gossip.channel] reportMembershipChanges -> INFO d20b Membership view has changed. peers went offline: [[peer0.org18.example.com:7051 ]] , current view: [[peer0.org8.example.com:7051 ] [peer0.org17.example.com:7051 ] [peer0.org1.example.com:7051 ] [peer0.org12.example.com:7051 ] [peer0.org13.example.com:7051 ] [peer0.org11.example.com:7051 ] [peer0.org19.example.com:7051 ] [peer0.org10.example.com:7051 ] [peer0.org14.example.com:7051 ] [peer0.org15.example.com:7051 ] [peer0.org6.example.com:7051 ] [peer0.org7.example.com:7051 ] [peer0.org23.example.com:7051 ] [peer0.org21.example.com:7051 ] [peer0.org16.example.com:7051 ] [peer0.org5.example.com:7051 ] [peer0.org9.example.com:7051 ] [peer0.org4.example.com:7051 ] [peer0.org20.example.com:7051 ] [peer0.org22.example.com:7051 ]]```
Fabric network setup: 23 Organisations. They are hosted on 2 VMs on a docker swarm legacy setup. There will be 1 peer per org (23 peers total) and 1 ca per org (23 CAs total). There is a SOLO orderer setup, with only one channel that every org and peer will join. There are 2 chaincodes instantiated on the channel, and both will be installed on every peer. One of the chaincodes use private data collections. Our VMs are 16G CPU and 32GB Memory each.
mistogans (Thu, 25 Apr 2019 01:57:11 GMT):
Hi, we are seeing problems with the gossip membership list on every peer in our Fabric network. In summary, the membership list is very unstable and peers constantly drop out of each other's list every few seconds. Below is a snippet of our logs from one of the peers.
```2019-04-25 01:51:55.608 UTC [gossip.discovery] expireDeadMembers -> WARN d202 Exiting
2019-04-25 01:51:59.140 UTC [gossip.channel] reportMembershipChanges -> INFO d203 Membership view has changed. peers went offline: [[peer0.org11.example.com:7051 ]] , current view: [[peer0.org10.example.com:7051 ] [peer0.org13.example.com:7051 ] [peer0.org19.example.com:7051 ] [peer0.org6.example.com:7051 ] [peer0.org14.example.com:7051 ] [peer0.org15.example.com:7051 ] [peer0.org4.example.com:7051 ] [peer0.org7.example.com:7051 ] [peer0.org23.example.com:7051 ] [peer0.org18.example.com:7051 ] [peer0.org21.example.com:7051 ] [peer0.org16.example.com:7051 ] [peer0.org5.example.com:7051 ] [peer0.org9.example.com:7051 ] [peer0.org22.example.com:7051 ] [peer0.org20.example.com:7051 ] [peer0.org12.example.com:7051 ] [peer0.org8.example.com:7051 ] [peer0.org17.example.com:7051 ] [peer0.org1.example.com:7051 ]]
2019-04-25 01:52:05.712 UTC [comm.grpc.server] 1 -> INFO d204 unary call completed {"grpc.start_time": "2019-04-25T01:52:05.712Z", "grpc.service": "gossip.Gossip", "grpc.method": "Ping", "grpc.request_deadline": "2019-04-25T01:52:07.712Z", "grpc.peer_address": "10.0.9.41:44594", "grpc.peer_subject": "CN=peer0.org22.example.com,L=San Francisco,ST=California,C=US", "grpc.code": "OK", "grpc.call_duration": "157.975µs"}
2019-04-25 01:52:09.140 UTC [gossip.channel] reportMembershipChanges -> INFO d205 Membership view has changed. peers went online: [[peer0.org11.example.com:7051 ]] , current view: [[peer0.org15.example.com:7051 ] [peer0.org6.example.com:7051 ] [peer0.org14.example.com:7051 ] [peer0.org23.example.com:7051 ] [peer0.org18.example.com:7051 ] [peer0.org21.example.com:7051 ] [peer0.org16.example.com:7051 ] [peer0.org5.example.com:7051 ] [peer0.org9.example.com:7051 ] [peer0.org4.example.com:7051 ] [peer0.org7.example.com:7051 ] [peer0.org20.example.com:7051 ] [peer0.org22.example.com:7051 ] [peer0.org17.example.com:7051 ] [peer0.org1.example.com:7051 ] [peer0.org12.example.com:7051 ] [peer0.org8.example.com:7051 ] [peer0.org11.example.com:7051 ] [peer0.org19.example.com:7051 ] [peer0.org10.example.com:7051 ] [peer0.org13.example.com:7051 ]]
2019-04-25 01:52:09.353 UTC [comm.grpc.server] 1 -> INFO d206 streaming call completed {"grpc.start_time": "2019-04-25T01:49:29.299Z", "grpc.service": "gossip.Gossip", "grpc.method": "GossipStream", "grpc.peer_address": "10.0.9.3:39924", "grpc.peer_subject": "CN=peer0.org13.example.com,L=San Francisco,ST=California,C=US", "error": "rpc error: code = Canceled desc = context canceled", "grpc.code": "Canceled", "grpc.call_duration": "2m40.05450849s"}
2019-04-25 01:52:13.110 UTC [gossip.discovery] getDeadMembers -> WARN d207 Haven't heard from [228 6 93 182 126 13 90 123 11 24 161 26 85 17 97 232 130 132 194 172 229 28 198 59 93 41 219 118 182 70 42 5] for 25.629033151s
2019-04-25 01:52:13.113 UTC [gossip.discovery] expireDeadMembers -> WARN d208 Entering [e4065db67e0d5a7b0b18a11a551161e88284c2ace51cc63b5d29db76b6462a05]
2019-04-25 01:52:13.113 UTC [gossip.discovery] expireDeadMembers -> WARN d209 Closing connection to Endpoint: peer0.org18.example.com:7051, InternalEndpoint: , PKI-ID: e4065db67e0d5a7b0b18a11a551161e88284c2ace51cc63b5d29db76b6462a05, Metadata:
2019-04-25 01:52:13.114 UTC [gossip.discovery] expireDeadMembers -> WARN d20a Exiting
2019-04-25 01:52:14.140 UTC [gossip.channel] reportMembershipChanges -> INFO d20b Membership view has changed. peers went offline: [[peer0.org18.example.com:7051 ]] , current view: [[peer0.org8.example.com:7051 ] [peer0.org17.example.com:7051 ] [peer0.org1.example.com:7051 ] [peer0.org12.example.com:7051 ] [peer0.org13.example.com:7051 ] [peer0.org11.example.com:7051 ] [peer0.org19.example.com:7051 ] [peer0.org10.example.com:7051 ] [peer0.org14.example.com:7051 ] [peer0.org15.example.com:7051 ] [peer0.org6.example.com:7051 ] [peer0.org7.example.com:7051 ] [peer0.org23.example.com:7051 ] [peer0.org21.example.com:7051 ] [peer0.org16.example.com:7051 ] [peer0.org5.example.com:7051 ] [peer0.org9.example.com:7051 ] [peer0.org4.example.com:7051 ] [peer0.org20.example.com:7051 ] [peer0.org22.example.com:7051 ]]```
Fabric network setup: 23 Organisations. They are hosted on 2 VMs on a docker swarm legacy setup. There will be 1 peer per org (23 peers total) and 1 ca per org (23 CAs total). There is a SOLO orderer setup, with only one channel that every org and peer will join. There are 2 chaincodes instantiated on the channel, and both will be installed on every peer. One of the chaincodes use private data collections. Our VMs are 16G CPU and 32GB Memory each.
mistogans (Thu, 25 Apr 2019 01:57:11 GMT):
Hi, we are seeing problems with the gossip membership list on every peer in our Fabric network. In summary, the membership list is very unstable and peers constantly drop out of each other's list every few seconds. Below is a snippet of our logs from one of the peers. This happens even when the network is idle, and won't stop until I bring down the network.
```2019-04-25 01:51:55.608 UTC [gossip.discovery] expireDeadMembers -> WARN d202 Exiting
2019-04-25 01:51:59.140 UTC [gossip.channel] reportMembershipChanges -> INFO d203 Membership view has changed. peers went offline: [[peer0.org11.example.com:7051 ]] , current view: [[peer0.org10.example.com:7051 ] [peer0.org13.example.com:7051 ] [peer0.org19.example.com:7051 ] [peer0.org6.example.com:7051 ] [peer0.org14.example.com:7051 ] [peer0.org15.example.com:7051 ] [peer0.org4.example.com:7051 ] [peer0.org7.example.com:7051 ] [peer0.org23.example.com:7051 ] [peer0.org18.example.com:7051 ] [peer0.org21.example.com:7051 ] [peer0.org16.example.com:7051 ] [peer0.org5.example.com:7051 ] [peer0.org9.example.com:7051 ] [peer0.org22.example.com:7051 ] [peer0.org20.example.com:7051 ] [peer0.org12.example.com:7051 ] [peer0.org8.example.com:7051 ] [peer0.org17.example.com:7051 ] [peer0.org1.example.com:7051 ]]
2019-04-25 01:52:05.712 UTC [comm.grpc.server] 1 -> INFO d204 unary call completed {"grpc.start_time": "2019-04-25T01:52:05.712Z", "grpc.service": "gossip.Gossip", "grpc.method": "Ping", "grpc.request_deadline": "2019-04-25T01:52:07.712Z", "grpc.peer_address": "10.0.9.41:44594", "grpc.peer_subject": "CN=peer0.org22.example.com,L=San Francisco,ST=California,C=US", "grpc.code": "OK", "grpc.call_duration": "157.975µs"}
2019-04-25 01:52:09.140 UTC [gossip.channel] reportMembershipChanges -> INFO d205 Membership view has changed. peers went online: [[peer0.org11.example.com:7051 ]] , current view: [[peer0.org15.example.com:7051 ] [peer0.org6.example.com:7051 ] [peer0.org14.example.com:7051 ] [peer0.org23.example.com:7051 ] [peer0.org18.example.com:7051 ] [peer0.org21.example.com:7051 ] [peer0.org16.example.com:7051 ] [peer0.org5.example.com:7051 ] [peer0.org9.example.com:7051 ] [peer0.org4.example.com:7051 ] [peer0.org7.example.com:7051 ] [peer0.org20.example.com:7051 ] [peer0.org22.example.com:7051 ] [peer0.org17.example.com:7051 ] [peer0.org1.example.com:7051 ] [peer0.org12.example.com:7051 ] [peer0.org8.example.com:7051 ] [peer0.org11.example.com:7051 ] [peer0.org19.example.com:7051 ] [peer0.org10.example.com:7051 ] [peer0.org13.example.com:7051 ]]
2019-04-25 01:52:09.353 UTC [comm.grpc.server] 1 -> INFO d206 streaming call completed {"grpc.start_time": "2019-04-25T01:49:29.299Z", "grpc.service": "gossip.Gossip", "grpc.method": "GossipStream", "grpc.peer_address": "10.0.9.3:39924", "grpc.peer_subject": "CN=peer0.org13.example.com,L=San Francisco,ST=California,C=US", "error": "rpc error: code = Canceled desc = context canceled", "grpc.code": "Canceled", "grpc.call_duration": "2m40.05450849s"}
2019-04-25 01:52:13.110 UTC [gossip.discovery] getDeadMembers -> WARN d207 Haven't heard from [228 6 93 182 126 13 90 123 11 24 161 26 85 17 97 232 130 132 194 172 229 28 198 59 93 41 219 118 182 70 42 5] for 25.629033151s
2019-04-25 01:52:13.113 UTC [gossip.discovery] expireDeadMembers -> WARN d208 Entering [e4065db67e0d5a7b0b18a11a551161e88284c2ace51cc63b5d29db76b6462a05]
2019-04-25 01:52:13.113 UTC [gossip.discovery] expireDeadMembers -> WARN d209 Closing connection to Endpoint: peer0.org18.example.com:7051, InternalEndpoint: , PKI-ID: e4065db67e0d5a7b0b18a11a551161e88284c2ace51cc63b5d29db76b6462a05, Metadata:
2019-04-25 01:52:13.114 UTC [gossip.discovery] expireDeadMembers -> WARN d20a Exiting
2019-04-25 01:52:14.140 UTC [gossip.channel] reportMembershipChanges -> INFO d20b Membership view has changed. peers went offline: [[peer0.org18.example.com:7051 ]] , current view: [[peer0.org8.example.com:7051 ] [peer0.org17.example.com:7051 ] [peer0.org1.example.com:7051 ] [peer0.org12.example.com:7051 ] [peer0.org13.example.com:7051 ] [peer0.org11.example.com:7051 ] [peer0.org19.example.com:7051 ] [peer0.org10.example.com:7051 ] [peer0.org14.example.com:7051 ] [peer0.org15.example.com:7051 ] [peer0.org6.example.com:7051 ] [peer0.org7.example.com:7051 ] [peer0.org23.example.com:7051 ] [peer0.org21.example.com:7051 ] [peer0.org16.example.com:7051 ] [peer0.org5.example.com:7051 ] [peer0.org9.example.com:7051 ] [peer0.org4.example.com:7051 ] [peer0.org20.example.com:7051 ] [peer0.org22.example.com:7051 ]]```
Fabric network setup: 23 Organisations. They are hosted on 2 VMs on a docker swarm legacy setup. There will be 1 peer per org (23 peers total) and 1 ca per org (23 CAs total). There is a SOLO orderer setup, with only one channel that every org and peer will join. There are 2 chaincodes instantiated on the channel, and both will be installed on every peer. One of the chaincodes use private data collections. Our VMs are 16G CPU and 32GB Memory each.
mistogans (Thu, 25 Apr 2019 02:04:55 GMT):
Htop logs - April 25, 2019 10:04 AM
mistogans (Thu, 25 Apr 2019 02:05:05 GMT):
Originally we thought it was a CPU/Memory resource issue but htop seems to show that our VMs are pretty stable, here is a htop from one of our VMs
(Not sure how to put a file here, if anyone can help out)
Fias (Fri, 26 Apr 2019 06:53:41 GMT):
Has joined the channel.
spacemandev (Mon, 06 May 2019 18:49:20 GMT):
Has left the channel.
kantipov (Wed, 08 May 2019 09:44:42 GMT):
Has joined the channel.
IgorSim (Mon, 20 May 2019 10:15:57 GMT):
@mistogans does any of peer that went offline reappears online again?
IgorSim (Mon, 20 May 2019 10:15:57 GMT):
@mistogans does any peer(s) that went offline reappears online again?
gauthampamu (Mon, 20 May 2019 16:14:03 GMT):
We are working on a deployment and we are getting this message. Received AliveMessage from a peer with the same PKI-ID as myself. I had searched for this message and it must be coming from discovery_impl.go.
gauthampamu (Mon, 20 May 2019 16:14:03 GMT):
@dave.enyeart We are working on a deployment and we are getting this message. Received AliveMessage from a peer with the same PKI-ID as myself. I had searched for this message and it must be coming from discovery_impl.go.
gauthampamu (Mon, 20 May 2019 16:14:55 GMT):
Can you explain what cause these errors. In our deployment we have two peers in the same organization and the team configured both peers as anchor peers. I am thinking that is causing the error.
risc (Mon, 20 May 2019 20:11:46 GMT):
Has joined the channel.
yacovm (Mon, 20 May 2019 22:03:24 GMT):
@gauthampamu you used the same certificate for 2 different peers
yacovm (Mon, 20 May 2019 22:04:10 GMT):
each peer should have its own certificate....
gauthampamu (Mon, 20 May 2019 23:49:12 GMT):
@yacovm Thanks @yacovm for the response. We realized it after sending the message. We already changed it and the issue is resolved now.
scottz (Tue, 21 May 2019 17:20:38 GMT):
Could a gossip expert please offer an explanation to the question about anchor peer updates, discussed in https://jira.hyperledger.org/browse/FAB-15460 and comments? Thanks!
yacovm (Tue, 21 May 2019 17:55:47 GMT):
yes @scottz will comment in few min :)
swetha (Wed, 22 May 2019 16:34:43 GMT):
Has joined the channel.
swetha (Fri, 24 May 2019 19:58:30 GMT):
@MHBauer and I have started this document to keep track/discuss our fixes for the gossip flakes: https://docs.google.com/document/d/1ABJQ7F5ej-D16LzXmzICR3Prs5hoUNuZyvDy7u6GU8s/edit?usp=sharing /cc @sykesm @ronenschafferibm
ronenschafferibm (Fri, 24 May 2019 19:58:31 GMT):
Has joined the channel.
MHBauer (Fri, 24 May 2019 20:12:05 GMT):
we might switch it to a wiki page if we find the appropriate place...
swetha (Fri, 24 May 2019 23:32:36 GMT):
Created a wiki page: https://wiki.hyperledger.org/x/EoXT
risc (Fri, 31 May 2019 15:33:43 GMT):
Hi Guys, my customer wants to not use gossip between peer in different organisations, so 2 questions: 1- there is any other impacts besides Services Discovery and Private Date Collections? 2 - What is the best practices to disable it?
yacovm (Fri, 31 May 2019 16:30:25 GMT):
what's the rational behind disabling it?
risc (Fri, 31 May 2019 16:57:05 GMT):
They want to reduce the network (firewalls, ips etc) complexity
yacovm (Fri, 31 May 2019 16:57:52 GMT):
ok so just don't put any anchor peers or bootstrap peers
risc (Fri, 31 May 2019 17:07:09 GMT):
Anchors are just used for Gossip, correct?
risc (Fri, 31 May 2019 17:07:28 GMT):
Anchor are just used for gossip rigth?
risc (Fri, 31 May 2019 17:07:28 GMT):
Anchor are just used for gossip right?
risc (Fri, 31 May 2019 17:08:39 GMT):
and internally in the Org? can the peers use gossip to sync?
yacovm (Fri, 31 May 2019 18:28:01 GMT):
anchor peers are used to establish communication between orgs
risc (Fri, 31 May 2019 21:09:38 GMT):
great. Thx
MHBauer (Tue, 04 Jun 2019 23:29:46 GMT):
in gossip/state, what is antiEntropy ?
MHBauer (Tue, 04 Jun 2019 23:33:15 GMT):
what does the term mean or where is it described?
C0rWin (Wed, 05 Jun 2019 05:43:28 GMT):
antiEntropy used to sync state, e.g. bring block what outside of gossip push/pull sliding window
C0rWin (Wed, 05 Jun 2019 07:16:22 GMT):
described here: ftp://ftp.uk.i-scream.org/sites/www.bitsavers.org/pdf/xerox/parc/techReports/CSL-89-1_Epidemic_Algorithms_for_Replicated_Database_Maintenance.pdf
C0rWin (Wed, 05 Jun 2019 07:16:22 GMT):
@MHBauer what are you trying to do w/ it?
MHBauer (Wed, 05 Jun 2019 18:13:40 GMT):
understand what it is, why it has that name, what the name means. sort of a 5whys and answer the five w's kind of thing.
MHBauer (Wed, 05 Jun 2019 18:15:07 GMT):
and now that I have a paper, I can make sure it's correct and put a link to the paper somewhere.
MHBauer (Wed, 05 Jun 2019 18:26:43 GMT):
it's a very curious name. eye-catching
C0rWin (Thu, 06 Jun 2019 12:15:08 GMT):
> and now that I have a paper, I can make sure it's correct and put a link to the paper somewhere.
paper explains the context and why you cannot avoid it, while algorithmic wise it's not described.
while making sure it's correct and before publishing a link could you please keep me posted?
this is very general paper which kind of incepted all epidemic style protocols, while it has really little connection to what we have in Fabric.
hence I do not think publishing this link stating it's related to Fabric will be just wrong.
MHBauer (Thu, 06 Jun 2019 17:18:02 GMT):
okay, will keep posted, haven't started on either yet, was more of a curiosity issue at the time.
huxd (Wed, 14 Aug 2019 01:07:58 GMT):
Has joined the channel.
iamdm (Wed, 14 Aug 2019 08:59:05 GMT):
Has joined the channel.
tennenjl (Fri, 23 Aug 2019 19:58:58 GMT):
Dummy question.. Is there any validation of transactions when an offline peer comes back online and uses gossip to catch up?
yacovm (Fri, 23 Aug 2019 20:20:02 GMT):
@tennenjl - it's via the same mechanism...
yacovm (Fri, 23 Aug 2019 20:20:26 GMT):
gossip just pushes transactions into validation and then into the ledger
tennenjl (Fri, 23 Aug 2019 20:21:04 GMT):
Thanks so does that mean each peer will validate the transactions before they get recorded into the ledger.
yacovm (Fri, 23 Aug 2019 20:21:16 GMT):
of course
yacovm (Fri, 23 Aug 2019 20:21:25 GMT):
that's the whole point of Blockchain, no?
yacovm (Fri, 23 Aug 2019 20:21:30 GMT):
each peer to itself ;)
tennenjl (Fri, 23 Aug 2019 20:21:34 GMT):
100000% perecent
tennenjl (Fri, 23 Aug 2019 20:21:34 GMT):
100000% percent
soumyanayak (Mon, 26 Aug 2019 07:48:19 GMT):
Has joined the channel.
soumyanayak (Mon, 26 Aug 2019 07:49:01 GMT):
Hi Team
I inserted around 21 million records of data into blockchain creating around 2180 blocks across two peers .. so the main anchor peer - peer1 worked well having all the data
but the peer2 after aroud 13 million records it was throwing some error and and did not insert any more
i recorded the whole peer2 logs.
soumyanayak (Mon, 26 Aug 2019 07:49:22 GMT):
peer2.zip
soumyanayak (Mon, 26 Aug 2019 07:51:06 GMT):
Attached is the peer2.zip.
i tried the second time -- by removing the peer2 ledgers and making it up to again sync up data from peer1 but only after 1.6 million it started throwing the below issue
2019-08-26 07:40:36.912 UTC [gossip.gossip] handleMessage -> DEBU bbe Exiting
2019-08-26 07:40:36.918 UTC [gossip.pull] Hello -> DEBU bbf Sending BLOCK_MSG hello to 10.65.192.148:7051
2019-08-26 07:40:36.918 UTC [gossip.comm] Send -> DEBU bc0 Entering, sending GossipMessage: channel:"legaldescriptionchannel" tag:CHAN_AND_ORG hello:
soumyanayak (Mon, 26 Aug 2019 07:52:16 GMT):
the network is good -- Fabric - 1.4.2v, kafka-zookeeper based orderer, Peers running in docker containers in azure VMs, 1 TB of space available for both peers
soumyanayak (Mon, 26 Aug 2019 07:52:25 GMT):
Please help and advice
yacovm (Mon, 26 Aug 2019 11:22:02 GMT):
@soumyanayak seems like the blocks are too big
yacovm (Mon, 26 Aug 2019 11:22:36 GMT):
decrease https://github.com/hyperledger/fabric/blob/master/sampleconfig/core.yaml#L226
soumyanayak (Mon, 26 Aug 2019 11:46:52 GMT):
Which parameter i have to decrease and to what value
state:
# indicates whenever state transfer is enabled or not
# default value is true, i.e. state transfer is active
# and takes care to sync up missing blocks allowing
# lagging peer to catch up to speed with rest network
enabled: true
# checkInterval interval to check whether peer is lagging behind enough to
# request blocks via state transfer from another peer.
checkInterval: 10s
# responseTimeout amount of time to wait for state transfer response from
# other peers
responseTimeout: 3s
# batchSize the number of blocks to request via state transfer from another peer
batchSize: 10
# blockBufferSize reflects the size of the re-ordering buffer
# which captures blocks and takes care to deliver them in order
# down to the ledger layer. The actually buffer size is bounded between
# 0 and 2*blockBufferSize, each channel maintains its own buffer
blockBufferSize: 100
# maxRetries maximum number of re-tries to ask
# for single state transfer request
maxRetries: 3
yacovm (Mon, 26 Aug 2019 11:47:31 GMT):
the parameter in my link
yacovm (Mon, 26 Aug 2019 11:47:43 GMT):
batchSize
soumyanayak (Mon, 26 Aug 2019 11:48:56 GMT):
what is the value i should put and how we decide that value like based on which parameter?
soumyanayak (Mon, 26 Aug 2019 11:55:29 GMT):
what is the value i should put and how we decide that value like based on which parameter?
yacovm (Mon, 26 Aug 2019 11:56:11 GMT):
just put 5 and try
soumyanayak (Mon, 26 Aug 2019 12:14:18 GMT):
So is there any limit on the total no of size of data("n" no of blocks) to be gossiped on a channel and if so whats that max size?
yacovm (Mon, 26 Aug 2019 12:14:37 GMT):
unfortunately it's per block count and not per MB
yacovm (Mon, 26 Aug 2019 12:14:48 GMT):
there is a limit on the total size of gRPC messages
yacovm (Mon, 26 Aug 2019 12:14:51 GMT):
it's 100MB I think
soumyanayak (Mon, 26 Aug 2019 12:15:30 GMT):
means lets say mine each block is around 20 MB so i can have the batchSize as 5 right of more than that then it would fail?
soumyanayak (Mon, 26 Aug 2019 12:15:30 GMT):
means lets say mine each block is around 20 MB so i can have the batchSize as 5 right ? if more than that then it would fail crossing the 100 limit of GRPC message?
yacovm (Mon, 26 Aug 2019 12:17:55 GMT):
then put 4 ;)
yacovm (Mon, 26 Aug 2019 12:17:58 GMT):
to be safe
soumyanayak (Mon, 26 Aug 2019 12:19:40 GMT):
So how to know in that case as blocks may be of varying sizes .. it might happen let say given batchSize as 4 and the total size is less than 100 mb then it goes through , if at all it happens the next 4 blocks is of 120 MB then it would fail at that moment and would never proceed ? So how this would be handled ?
soumyanayak (Mon, 26 Aug 2019 12:22:01 GMT):
that 100MB is depicted by this property - blockBufferSize: 100 ?
yacovm (Mon, 26 Aug 2019 12:22:52 GMT):
no that block buffer size is unrelated
soumyanayak (Mon, 26 Aug 2019 12:26:41 GMT):
Ok but how to handle the blocks of varying sizes when there will be bulk transactions happening
yacovm (Mon, 26 Aug 2019 12:26:57 GMT):
you can limit the size of the block in the orderer
yacovm (Mon, 26 Aug 2019 12:27:03 GMT):
and thus ensure the blocks are not too large
soumyanayak (Mon, 26 Aug 2019 12:30:29 GMT):
where we do that in orderer are you telling we need to do in configtx.yaml ?
BatchSize:
MaxMessageCount: 10
AbsoluteMaxBytes: 98 MB
PreferredMaxBytes: 512 KB
or in any other parameter ?
yacovm (Mon, 26 Aug 2019 12:31:04 GMT):
preferred max bytes
soumyanayak (Mon, 26 Aug 2019 12:33:26 GMT):
so a question here is then by default if the value is 98 or 99 MB max then it should be the case that in peer - core.yaml the batchSize should be by default 1 right instead of 10 . Based on the preferred max bytes we reduce we increase the batchSize in core.yaml ?
yacovm (Mon, 26 Aug 2019 12:34:27 GMT):
i don't understand
soumyanayak (Mon, 26 Aug 2019 12:37:27 GMT):
my question was the maximum size of the block can be upto 98 MB based on the absoluteMaxBytes. So based on that if a block goes upto 98 MB and the GRPC size allowed is upto 100 MB , then the default size of the pramater batchSize in core.yaml should be by default 1 right.
soumyanayak (Mon, 26 Aug 2019 12:38:17 GMT):
in my case i am putting 10000 rows of data in one block
yacovm (Mon, 26 Aug 2019 12:38:18 GMT):
yeah it can.... but you want to make it *smaller* no?
soumyanayak (Mon, 26 Aug 2019 12:38:28 GMT):
yeah
yacovm (Mon, 26 Aug 2019 12:38:31 GMT):
why do you put so much data in the block?
soumyanayak (Mon, 26 Aug 2019 12:39:53 GMT):
actually i have around 35 millions of rows of data which has to be back filled from a sql server to the blockchain and then the transactions would happen on the data -- so in a loop we were trying to run and insert 10000 rows of data to one block...
soumyanayak (Mon, 26 Aug 2019 12:40:05 GMT):
is this the ideal way of doing or any other way you can suggest
soumyanayak (Mon, 26 Aug 2019 12:41:07 GMT):
each row of data is having two fields that is inserted
soumyanayak (Mon, 26 Aug 2019 12:43:57 GMT):
so each block was coming around 20-25 MB i had fetched one block and checked
yacovm (Mon, 26 Aug 2019 12:45:01 GMT):
do it in several transactions
soumyanayak (Mon, 26 Aug 2019 13:19:01 GMT):
ok if i put the value in peer.state.enabled as false in core.yaml .. so what would be the default batchSize it would take?
yacovm (Mon, 26 Aug 2019 13:20:47 GMT):
if you disable the state transfer then there is no point to point bkicj replication
yacovm (Mon, 26 Aug 2019 13:20:49 GMT):
*block
soumyanayak (Mon, 26 Aug 2019 13:22:19 GMT):
ok so it means that it will never sync up with any other bootstrap peer for the data .. so it should be mandatory enabled right?
yacovm (Mon, 26 Aug 2019 13:22:41 GMT):
it can sync from orderer
soumyanayak (Mon, 26 Aug 2019 13:24:23 GMT):
when it does from orderer what would be the batch size or no of blocks fetched at each call ? WIll there be any performance difference fetching from the orderer or from other bootstrapped peers . Wanted to know the best practice here
soumyanayak (Mon, 26 Aug 2019 14:17:00 GMT):
any thoughts on this @yacovm
yacovm (Mon, 26 Aug 2019 14:17:15 GMT):
no, i'm too busy with other stuff atm
soumyanayak (Mon, 26 Aug 2019 15:29:33 GMT):
@yacovm i tried updating from orderer directly .. so what ia observed was the whole azure VM memory is getting used up and the system is becoming very slow even to SSH into it. So is there any reason of the whole memory getting used up or any way to handle that?
yacovm (Mon, 26 Aug 2019 15:29:51 GMT):
yeah of course
yacovm (Mon, 26 Aug 2019 15:30:02 GMT):
how much memory do you have?
yacovm (Mon, 26 Aug 2019 15:30:03 GMT):
in the VMs?
soumyanayak (Mon, 26 Aug 2019 15:30:07 GMT):
8 Gb RAM
soumyanayak (Mon, 26 Aug 2019 15:31:34 GMT):
Is this an ideal memory to have ?
yacovm (Mon, 26 Aug 2019 16:22:02 GMT):
dunno seems enough
soumyanayak (Mon, 09 Sep 2019 09:45:29 GMT):
@yacovm is there any plan to have a API or CLI command to check whether a given peer is an anchor peer or not? Any JIRA raised for it
yacovm (Mon, 09 Sep 2019 09:46:34 GMT):
just look at the latest config block
yacovm (Mon, 09 Sep 2019 09:46:37 GMT):
it should be there
soumyanayak (Mon, 09 Sep 2019 09:48:32 GMT):
ok
soumyanayak (Tue, 01 Oct 2019 06:08:41 GMT):
when tried joining a peer to the channel its giving the below error
peer1 | 2019-10-01 06:06:29.053 UTC [gossip.service] updateAnchors -> ERRO 321 Tried joining channel legaldescriptionchannel but our org( OrgA ), isn't among the orgs of the channel: [Org1] , aborting.
What does it mean
yacovm (Tue, 01 Oct 2019 08:05:52 GMT):
@soumyanayak - it seems that the org OrgA was added to the channel legaldescriptionchannel later and was not there from the start, right?
soumyanayak (Tue, 01 Oct 2019 08:06:39 GMT):
yes @yacovm orgA was added later as part of config update
yacovm (Tue, 01 Oct 2019 08:06:44 GMT):
right
yacovm (Tue, 01 Oct 2019 08:06:47 GMT):
this is natural
soumyanayak (Tue, 01 Oct 2019 08:06:53 GMT):
i added one more orgB same issue for that also
yacovm (Tue, 01 Oct 2019 08:06:53 GMT):
ignore the error ;)
yacovm (Tue, 01 Oct 2019 08:07:02 GMT):
yeah, ignore the error then
yacovm (Tue, 01 Oct 2019 08:07:05 GMT):
it's fine
soumyanayak (Tue, 01 Oct 2019 08:07:10 GMT):
but i am getting it continuously
yacovm (Tue, 01 Oct 2019 08:07:25 GMT):
it will stop when the peer will get the blocks from the orderr
yacovm (Tue, 01 Oct 2019 08:07:27 GMT):
*orderer
soumyanayak (Tue, 01 Oct 2019 08:09:02 GMT):
Clipboard - October 1, 2019 1:38 PM
soumyanayak (Tue, 01 Oct 2019 08:09:17 GMT):
these are the logs going on is it normal then ?
soumyanayak (Tue, 01 Oct 2019 08:11:34 GMT):
Same thing is repeating every 1 minute
soumyanayak (Tue, 01 Oct 2019 08:12:54 GMT):
and when i am trying to do individual org anchor update for the newly added orgs i am getting the below error
Error: got unexpected status: FORBIDDEN -- implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Writers' sub-policies to be satisfied: permission denied
yacovm (Tue, 01 Oct 2019 08:15:21 GMT):
oh... if you get a forbidden error - it's bad
yacovm (Tue, 01 Oct 2019 08:15:29 GMT):
it means that your organization is not really in the channel
yacovm (Tue, 01 Oct 2019 08:15:29 GMT):
it means that your organization is not really in the channel or doesn't have permission to pull the blocks
yacovm (Tue, 01 Oct 2019 08:15:33 GMT):
you did something wrong :/
soumyanayak (Tue, 01 Oct 2019 08:17:36 GMT):
how we can check or debug where it was wrong like getting the latest config block and checking from the config JSON or how to do this?
yacovm (Tue, 01 Oct 2019 08:20:20 GMT):
oh check the tutorial of channel update
yacovm (Tue, 01 Oct 2019 08:20:35 GMT):
https://hyperledger-fabric.readthedocs.io/en/release-1.4/config_update.html
soumyanayak (Tue, 01 Oct 2019 09:25:22 GMT):
@yacovm i was able to make the two org peers join the channel -- so there was admin certificate issue
soumyanayak (Tue, 01 Oct 2019 09:25:27 GMT):
now the issues are not coming
soumyanayak (Tue, 01 Oct 2019 09:26:09 GMT):
but now when i am trying to update the anchor peer for both the orgs after generating the anchorupdate transaction file using configtxgen below is the issue thrown
soumyanayak (Tue, 01 Oct 2019 09:26:20 GMT):
Error: got unexpected status: BAD_REQUEST -- error applying config update to existing channel 'legaldescriptionchannel': error authorizing update: error validating ReadSet: proposed update requires that key [Group] /Channel/Application be at version 1, but it is currently at version 3
soumyanayak (Tue, 01 Oct 2019 09:26:20 GMT):
```Error: got unexpected status: BAD_REQUEST -- error applying config update to existing channel 'legaldescriptionchannel': error authorizing update: error validating ReadSet: proposed update requires that key [Group] /Channel/Application be at version 1, but it is currently at version 3```
soumyanayak (Tue, 01 Oct 2019 09:28:40 GMT):
and for the other organisation its giving the below error
soumyanayak (Tue, 01 Oct 2019 09:28:42 GMT):
Error: got unexpected status: BAD_REQUEST -- error applying config update to existing channel 'legaldescriptionchannel': error authorizing update: error validating ReadSet: proposed update requires that key [Group] /Channel/Application be at version 1, but it is currently at version 3
soumyanayak (Tue, 01 Oct 2019 09:28:42 GMT):
```Error: got unexpected status: BAD_REQUEST -- error applying config update to existing channel 'legaldescriptionchannel': error authorizing update: error validating ReadSet: proposed update requires that key [Group] /Channel/Application be at version 1, but it is currently at version 3```
soumyanayak (Tue, 01 Oct 2019 09:35:53 GMT):
Same issue for both-- how do we update the anchor peer in this case -- do we again go by the way of config update ?
soumyanayak (Tue, 01 Oct 2019 12:03:46 GMT):
Got the solution by config update need to update the anchors
jyxie2007 (Mon, 18 Nov 2019 04:00:06 GMT):
Has joined the channel.
ZainabM (Thu, 09 Jan 2020 11:53:48 GMT):
Has joined the channel.
Nammalvar (Thu, 23 Jan 2020 08:51:10 GMT):
Has joined the channel.
lzaouche (Tue, 28 Jan 2020 09:17:33 GMT):
Has joined the channel.
conanoc (Thu, 06 Feb 2020 07:11:11 GMT):
Has joined the channel.
conanoc (Thu, 06 Feb 2020 07:28:07 GMT):
Document say, "Note that in order for peers to be known to the discovery service, they must have an EXTERNAL_ENDPOINT defined. "
https://hyperledger-fabric.readthedocs.io/en/latest/discovery-overview.html#how-service-discovery-works-in-fabric
How can I define EXTERNAL_ENDPOINT in peer and orderer? Is it different from CORE_PEER_GOSSIP_EXTERNALENDPOINT?
yacovm (Thu, 06 Feb 2020 09:49:48 GMT):
it's only for the peer
yacovm (Thu, 06 Feb 2020 09:49:57 GMT):
and this is exactly what you wrote
yacovm (Thu, 06 Feb 2020 09:50:02 GMT):
`CORE_PEER_GOSSIP_EXTERNALENDPOINT`
conanoc (Fri, 07 Feb 2020 01:41:58 GMT):
@yacovm Then, How is it possible that orderer's addresses could be found by discovery service? And I could not find any reference(not any but nothing meaningful) of CORE_PEER_GOSSIP_EXTERNALENDPOINT from the source code of fabric. Where can I find the code handling the CORE_PEER_GOSSIP_EXTERNALENDPOINT?
yacovm (Fri, 07 Feb 2020 09:08:09 GMT):
Look for "externalEndpoint"
yacovm (Fri, 07 Feb 2020 09:08:34 GMT):
The discovery returns the addresses from the channel config block
Antimttr (Tue, 10 Mar 2020 20:04:12 GMT):
Has joined the channel.
Antimttr (Tue, 10 Mar 2020 20:24:54 GMT):
hello, having an issue after all my peers are joined to the channel and i add the anchor peers im getting these warnings in my gossip logs in the first anchor peer: ```
2020-03-10 20:24:03.947 UTC [gossip.comm] createConnection -> WARN ac5 Remote endpoint claims to be a different peer, expected 051881a63876d5f0fba6b9a7e533a7ccb5fb203536b6fcee1ad72343fcb408e8 but got 0c5e3f1de15afe0619397274fb51d35e0bfe90b48cc3c641daaf197ae587bf7c
2020-03-10 20:24:03.947 UTC [gossip.comm] sendToEndpoint -> WARN ac6 Failed obtaining connection for 3.1X.72.X:7011, PKIid:051881a63876d5f0fba6b9a7e533a7ccb5fb203536b6fcee1ad72343fcb408e8 reason: authentication failure
```
Antimttr (Tue, 10 Mar 2020 20:25:33 GMT):
so the socket in the second warning is the socket of the peer for which logs im following
Antimttr (Tue, 10 Mar 2020 20:25:42 GMT):
so im not sure how it makes sense that its failiing to connect to itself, but there it is
Antimttr (Tue, 10 Mar 2020 20:27:34 GMT):
so from the anchor peer in org2 i get these warnings: ```
2020-03-10 20:26:50.976 UTC [gossip.comm] createConnection -> WARN 803 Remote endpoint claims to be a different peer, expected 0c5e3f1de15afe0619397274fb51d35e0bfe90b48cc3c641daaf197ae587bf7c but got 051881a63876d5f0fba6b9a7e533a7ccb5fb203536b6fcee1ad72343fcb408e8
2020-03-10 20:26:50.976 UTC [gossip.comm] sendToEndpoint -> WARN 804 Failed obtaining connection for 3.X.72.X:7021, PKIid:0c5e3f1de15afe0619397274fb51d35e0bfe90b48cc3c641daaf197ae587bf7c reason: authentication failure
```
Antimttr (Tue, 10 Mar 2020 20:27:51 GMT):
so 7021 is the listening port of the anchor in org2 as well
Antimttr (Tue, 10 Mar 2020 20:28:01 GMT):
so its like they're trying to connect to themselves and failing?
Antimttr (Tue, 10 Mar 2020 20:29:38 GMT):
i have the anchor peer in org two setup with the following config parameters: ```
- CORE_PEER_LISTENADDRESS=0.0.0.0:7021
- CORE_PEER_GOSSIP_EXTERNALENDPOINT=3.X.72.X:7011
```
Antimttr (Tue, 10 Mar 2020 20:30:04 GMT):
so the second config entry should be the socket for the anchor peer of org1
Antimttr (Tue, 10 Mar 2020 20:30:07 GMT):
if im not mistaken
Antimttr (Tue, 10 Mar 2020 20:30:46 GMT):
why would it be trying to connect to itself though?
Antimttr (Tue, 10 Mar 2020 20:31:21 GMT):
also what are those big long strings where it says "expected"
Antimttr (Tue, 10 Mar 2020 20:31:31 GMT):
are those hashes of the ip address or something?
yacovm (Tue, 10 Mar 2020 21:00:38 GMT):
@Antimttr has the anchor peer joined the channel?
Antimttr (Tue, 10 Mar 2020 21:00:51 GMT):
yes ive got anchor peers from both orgs in the channel
Antimttr (Tue, 10 Mar 2020 21:00:59 GMT):
but ive been reading about those externalendpoints
Antimttr (Tue, 10 Mar 2020 21:01:02 GMT):
i might not even need them
yacovm (Tue, 10 Mar 2020 21:01:06 GMT):
ahh:
> 2020-03-10 20:24:03.947 UTC [gossip.comm] createConnection -> WARN ac5 Remote endpoint claims to be a different peer, expected 051881a63876d5f0fba6b9a7e533a7ccb5fb203536b6fcee1ad72343fcb408e8 but got 0c5e3f1de15afe0619397274fb51d35e0bfe90b48cc3c641daaf197ae587bf7c
yacovm (Tue, 10 Mar 2020 21:01:09 GMT):
that's the problem
Antimttr (Tue, 10 Mar 2020 21:01:28 GMT):
so i have the anchors configured with externalendpoint andi shouldn't correct?
yacovm (Tue, 10 Mar 2020 21:01:28 GMT):
so, i can tell you when it happens process wise
yacovm (Tue, 10 Mar 2020 21:01:38 GMT):
maybe it will give you a clue
yacovm (Tue, 10 Mar 2020 21:01:51 GMT):
basically anchor peer connection is a 2 staged process
yacovm (Tue, 10 Mar 2020 21:02:17 GMT):
consider yourself a peer that you get a new config block of a channel, or the genesis block of that channel
yacovm (Tue, 10 Mar 2020 21:02:23 GMT):
and you see you have anchor peers
yacovm (Tue, 10 Mar 2020 21:02:26 GMT):
so you do 2 things:
yacovm (Tue, 10 Mar 2020 21:02:44 GMT):
1) Probe the anchor peer to find out its certificate and organization association
yacovm (Tue, 10 Mar 2020 21:03:08 GMT):
2) Connect to the anchor peer again, and initiate the standard gossip membership exchange protocols and stuff
yacovm (Tue, 10 Mar 2020 21:03:24 GMT):
trick question: why would you want the first stage?
Antimttr (Tue, 10 Mar 2020 21:03:45 GMT):
hmm
Antimttr (Tue, 10 Mar 2020 21:04:08 GMT):
seems like exactly what you'd get with step 2
yacovm (Tue, 10 Mar 2020 21:04:35 GMT):
in step 2 you also acquire the certificate as well
yacovm (Tue, 10 Mar 2020 21:04:59 GMT):
so the answer is - if somehow, someone took over the network routing
Antimttr (Tue, 10 Mar 2020 21:05:02 GMT):
are both steps done over gossip or is one done via grpc?
yacovm (Tue, 10 Mar 2020 21:05:11 GMT):
no, it's 2 different TLS connections
yacovm (Tue, 10 Mar 2020 21:05:34 GMT):
the reason is - if someone took over the routing and redirects all the communication into itself
yacovm (Tue, 10 Mar 2020 21:05:42 GMT):
or some stupid admin
yacovm (Tue, 10 Mar 2020 21:05:49 GMT):
configued the wrong IP for anchor peers
yacovm (Tue, 10 Mar 2020 21:05:56 GMT):
that doesn't correspond do its own organizatio
yacovm (Tue, 10 Mar 2020 21:06:18 GMT):
you don't want peers to connect to the wrong IP (could be a peer from a malicious organization or just some hacker's server)
yacovm (Tue, 10 Mar 2020 21:06:26 GMT):
so you first "sniff" the remote peer
yacovm (Tue, 10 Mar 2020 21:06:29 GMT):
but don't tell it anything
Antimttr (Tue, 10 Mar 2020 21:06:32 GMT):
i see
yacovm (Tue, 10 Mar 2020 21:06:38 GMT):
later, at the second stage
yacovm (Tue, 10 Mar 2020 21:06:45 GMT):
after you acquire the certificate from the first stage
yacovm (Tue, 10 Mar 2020 21:07:04 GMT):
you expect the 2nd connection to still have the same certificate from the first stage
yacovm (Tue, 10 Mar 2020 21:07:15 GMT):
if you detect the certificate changed
yacovm (Tue, 10 Mar 2020 21:07:19 GMT):
then something is wrong here
Antimttr (Tue, 10 Mar 2020 21:07:31 GMT):
so how would my certificate have changed from first two second stage..
yacovm (Tue, 10 Mar 2020 21:07:33 GMT):
now obviously it's just a "software engineering" problem
yacovm (Tue, 10 Mar 2020 21:07:44 GMT):
because you could have done stage 1+2 together
yacovm (Tue, 10 Mar 2020 21:07:47 GMT):
but unfortunately
yacovm (Tue, 10 Mar 2020 21:07:56 GMT):
gossip communication layer is decoupled from the membership layer :(
yacovm (Tue, 10 Mar 2020 21:08:10 GMT):
so the communication code doesn't know what an organization, or an anchor peer even means
yacovm (Tue, 10 Mar 2020 21:08:42 GMT):
> so how would my certificate have changed from first two second stage..
Probably you restarted the peer or something
yacovm (Tue, 10 Mar 2020 21:08:54 GMT):
and you re-created the certificate
Antimttr (Tue, 10 Mar 2020 21:08:57 GMT):
these are both fresh peers
Antimttr (Tue, 10 Mar 2020 21:09:22 GMT):
i basically docker-compose rm -f all the peers and the orderer
yacovm (Tue, 10 Mar 2020 21:09:24 GMT):
i think the endpoint in the past belonged to some different organization
Antimttr (Tue, 10 Mar 2020 21:09:39 GMT):
then started them up and then joined them to the channel
Antimttr (Tue, 10 Mar 2020 21:09:47 GMT):
and as soon as i designated the two anchor peers
Antimttr (Tue, 10 Mar 2020 21:09:51 GMT):
i started getting these warnings
yacovm (Tue, 10 Mar 2020 21:10:04 GMT):
https://github.com/hyperledger/fabric/blob/master/gossip/comm/comm_impl.go#L176-L180
yacovm (Tue, 10 Mar 2020 21:10:09 GMT):
this is the code that prints it
Antimttr (Tue, 10 Mar 2020 21:11:26 GMT):
my anchor peers are specified in the genesis block
Antimttr (Tue, 10 Mar 2020 21:11:53 GMT):
with the same socket that im getting the error messages on
Antimttr (Tue, 10 Mar 2020 21:12:07 GMT):
what's confusing to me is that my anchor peer for org1: peer1.org1.example.com
Antimttr (Tue, 10 Mar 2020 21:12:37 GMT):
that anchor peers logs are `2020-03-10 21:12:20.637 UTC [gossip.comm] sendToEndpoint -> WARN 2204 Failed obtaining connection for 3.x.72.x:7011, PKIid:051881a63876d5f0fba6b9a7e533a7ccb5fb203536b6fcee1ad72343fcb408e8 reason: authentication failure
`
Antimttr (Tue, 10 Mar 2020 21:12:37 GMT):
that anchor peers logs are `2020-03-10 21:12:20.637 UTC [gossip.comm] sendToEndpoint -> WARN 2204 Failed obtaining connection for 3.x.72.x:7011, PKIid:051881a63876d5f0fba6b9a7e533a7ccb5fb203536b6fcee1ad72343fcb408e8 reason: authentication failure`
Antimttr (Tue, 10 Mar 2020 21:12:55 GMT):
that IS the socket for peer1.org1.example.com
Antimttr (Tue, 10 Mar 2020 21:13:08 GMT):
so why would the anchor peer try to connect to itself
Antimttr (Tue, 10 Mar 2020 21:13:33 GMT):
maybe i should decode the current config block on the channel
Antimttr (Tue, 10 Mar 2020 21:13:42 GMT):
and check to make sure that the anchor peers are what i think they are
yacovm (Tue, 10 Mar 2020 21:13:53 GMT):
anchor peers should not be specified in genesis blocks
Antimttr (Tue, 10 Mar 2020 21:13:57 GMT):
oh really?
yacovm (Tue, 10 Mar 2020 21:14:01 GMT):
yeah....
yacovm (Tue, 10 Mar 2020 21:14:10 GMT):
you need to do a subsequent config block for that
yacovm (Tue, 10 Mar 2020 21:14:15 GMT):
look at the samples
Antimttr (Tue, 10 Mar 2020 21:14:24 GMT):
i am basing this all off of balance-trasnfer
Antimttr (Tue, 10 Mar 2020 21:14:46 GMT):
im pretty srue thats in the configtx.yaml of the balance trasnfer example
Antimttr (Tue, 10 Mar 2020 21:14:48 GMT):
let me double check
yacovm (Tue, 10 Mar 2020 21:14:49 GMT):
look at first_network
Antimttr (Tue, 10 Mar 2020 21:15:34 GMT):
ok let me look at first_network
Antimttr (Tue, 10 Mar 2020 21:16:13 GMT):
oh this one has policies in it already
Antimttr (Tue, 10 Mar 2020 21:16:16 GMT):
i was looking for that
Antimttr (Tue, 10 Mar 2020 21:16:33 GMT):
```
# leave this flag set to true.
AnchorPeers:
# AnchorPeers defines the location of peers which can be used
# for cross org gossip communication. Note, this value is only
# encoded in the genesis block in the Application section context
- Host: peer0.org1.example.com
Port: 7051
```
Antimttr (Tue, 10 Mar 2020 21:16:41 GMT):
so thats from configtx.yaml in first-network
Antimttr (Tue, 10 Mar 2020 21:16:48 GMT):
but you're saying i shouldnt have it in there?
yacovm (Tue, 10 Mar 2020 21:17:07 GMT):
https://github.com/hyperledger/fabric-samples/blob/master/first-network/scripts/utils.sh#L71-L90
yacovm (Tue, 10 Mar 2020 21:17:11 GMT):
look at this ^
yacovm (Tue, 10 Mar 2020 21:17:17 GMT):
this is how you do it
Antimttr (Tue, 10 Mar 2020 21:17:56 GMT):
ive been doing it after i join the channel with the JAVA SDK
Antimttr (Tue, 10 Mar 2020 21:18:09 GMT):
but i guess its kind of repetitive if i already have it in the genesis block
Antimttr (Tue, 10 Mar 2020 21:18:17 GMT):
but if i shouldnt have it in the genesis block at all
Antimttr (Tue, 10 Mar 2020 21:18:23 GMT):
then ill just remove it and do it after i join the channel
yacovm (Tue, 10 Mar 2020 21:19:05 GMT):
there is a reason behind it
yacovm (Tue, 10 Mar 2020 21:19:22 GMT):
you need to do it the "hard" way as i said
Antimttr (Tue, 10 Mar 2020 21:20:11 GMT):
or i guess its not the genesis block but the channel.tx ?
Antimttr (Tue, 10 Mar 2020 21:20:54 GMT):
ok ill take the anchor peer out of my configtx.yaml
Antimttr (Tue, 10 Mar 2020 21:21:14 GMT):
and then add it after i join the channel
yacovm (Tue, 10 Mar 2020 21:21:56 GMT):
yes
Antimttr (Tue, 10 Mar 2020 21:22:18 GMT):
ok thanks, ill give that a shot and report back here
Antimttr (Tue, 10 Mar 2020 21:23:17 GMT):
doing the anchor add with the javasdk should be equivilent to doing it with peer command, right?
Antimttr (Tue, 10 Mar 2020 21:25:15 GMT):
well ill try it with sdk if it doesnt work ill try it with cli
yacovm (Tue, 10 Mar 2020 21:27:55 GMT):
yes
Abhishekkishor (Thu, 12 Mar 2020 19:32:51 GMT):
Has joined the channel.
gokulalex (Fri, 20 Mar 2020 12:04:10 GMT):
Has joined the channel.
lehors (Tue, 31 Mar 2020 12:18:21 GMT):
Has left the channel.
matanyahu (Mon, 06 Apr 2020 17:45:23 GMT):
Has left the channel.
MHBauer (Thu, 16 Apr 2020 21:17:01 GMT):
Has left the channel.
conanoc (Tue, 19 May 2020 10:04:10 GMT):
Has left the channel.
dtomczyk (Thu, 11 Jun 2020 16:03:23 GMT):
Has joined the channel.
ViokingTung (Mon, 29 Jun 2020 08:37:49 GMT):
Has joined the channel.
ever-upwards (Tue, 07 Jul 2020 11:34:24 GMT):
Has joined the channel.
FarhanShafiq (Fri, 17 Jul 2020 09:10:59 GMT):
Has joined the channel.
FarhanShafiq (Fri, 17 Jul 2020 09:18:44 GMT):
Hello everyone, I have setup network of two orgs while each orgs have four anchor peers. I have started network with org1 and later add the org2 in the channel through updating channel block config.
When I installed chaincode on both organization peers then org2 giving me gossip privdata error.
I'm also getting endorsement error from each peer while invoking any function on chaincode.
`2020-07-17 09:14:35.818 UTC [gossip.privdata] fetchPrivateData -> DEBU 52b39 Total members in channel: []
2020-07-17 09:14:35.818 UTC [gossip.privdata] fetchPrivateData -> DEBU 52b3a Total members that fit some digest: []
2020-07-17 09:14:35.818 UTC [gossip.privdata] fetchPrivateData -> WARN 52b3b Do not know any peer in the channel( mychannel ) that matches the policies , aborting
2020-07-17 09:14:35.818 UTC [gossip.privdata] reconcile -> ERRO 52b3c reconciliation error when trying to fetch missing items from different peers: Empty membership
2020-07-17 09:14:35.818 UTC [gossip.privdata] run -> ERRO 52b3d Failed to reconcile missing private info, error: Empty membership`
xachen (Mon, 20 Jul 2020 13:48:44 GMT):
Has joined the channel.
xachen (Mon, 20 Jul 2020 13:48:45 GMT):
@guoger https://jira.hyperledger.org/browse/FAB-17035 do you meet this bug?
xachen (Mon, 20 Jul 2020 13:48:45 GMT):
@guoger https://jira.hyperledger.org/browse/FAB-17035 do you meet this bug? @yacovm
FarhanShafiq (Wed, 22 Jul 2020 08:31:05 GMT):
Hello, I'm trying to fetch peers from discover CLI but result giving me only target peer information. Is this the default behavior or should I expect all four peers information?
Note: I have set the "CORE_PEER_GOSSIP_EXTERNALENDPOINT" on all four anchor peers but still no way of knowing that gossip is running properly since, peers logs contain no warning and error.
dave.enyeart (Wed, 22 Jul 2020 11:59:33 GMT):
@FarhanShafiq Both this and the prior message indicate a gossip configuration problem. Please try with the test network that is known to have working gossip connection: https://hyperledger-fabric.readthedocs.io/en/latest/test_network.html And then you can compare your configuration with the working test network configuration.
FarhanShafiq (Wed, 22 Jul 2020 12:04:46 GMT):
Thanks @dave.enyeart for the reply, It was indeed problem of gossip configuration problem. First, I didn't provide the gossip bootstrap peers that's why peer was not able to discover other peers. Second, for adding anchor peers, I had to update the channel with anchor transaction.
FarhanShafiq (Wed, 22 Jul 2020 12:04:46 GMT):
Thanks @dave.enyeart for the reply, It was indeed problem of gossip configuration. First, I didn't provide the gossip bootstrap peers that's why peer was not able to discover other peers. Second, for adding anchor peers, I had to update the channel with anchor transaction.
FarhanShafiq (Wed, 22 Jul 2020 12:04:46 GMT):
Thanks @dave.enyeart for the reply, It was indeed problem of gossip configuration. First, I didn't provide the gossip bootstrap peers that's why peer was not able to discover other peers within the organizations. Second, for adding anchor peers, I had to update the channel with anchor transaction.
FarhanShafiq (Wed, 22 Jul 2020 12:40:58 GMT):
I bootstrapped my network with org1 and create and join mychannel after then, I updated config block trx to add the org2 and create and join mychannel from org2.
At this point, both organizations are able to discover peers within organization using `CORE_PEER_GOSSIP_BOOTSTRAP` but org1 peers are not able to discover org2 peers. For that, I updated the channel with org1 anchor trx for org2 peers to discover org1 peers but to my surprise, org1 peers are also able to discover org2 peers.
I know how org2 peers able to discover org1 peers using org1 anchor trx but how org1 peers are discovering org2 peers?
FarhanShafiq (Wed, 22 Jul 2020 12:43:13 GMT):
Let say, I add another org3 then org1 and org3 peers will be able discover each other? and what about org2 peers? Will org2 peers able to discover org3 peers?
FarhanShafiq (Wed, 22 Jul 2020 12:43:13 GMT):
Let say, I add org3 in mychannel then org1 and org3 peers will be able discover each other? and what about org2 peers? Will org2 peers able to discover org3 peers?
yacovm (Wed, 22 Jul 2020 14:00:34 GMT):
you need the peers to sync all blocks until the height where they were added
yacovm (Wed, 22 Jul 2020 14:00:47 GMT):
only then they can discover, because until then, gossip thinks they are not in the channel
FarhanShafiq (Wed, 22 Jul 2020 14:18:59 GMT):
Thanks. Existing org also will be able to discover new org peers without adding the anchor trx?
yacovm (Wed, 22 Jul 2020 15:18:32 GMT):
you just need a single anchor peer in a channel
yacovm (Wed, 22 Jul 2020 15:18:42 GMT):
but it's better if every org has its own anchor peers
FarhanShafiq (Wed, 22 Jul 2020 15:55:50 GMT):
Thanks, I have configured separate configtx.yaml for every org.
guptasndp10 (Sat, 19 Sep 2020 03:15:34 GMT):
Has joined the channel.
guptasndp10 (Sat, 19 Sep 2020 03:17:07 GMT):
@yacovm Could you please help me with one issue
guptasndp10 (Sat, 19 Sep 2020 03:19:28 GMT):
I have deployed a hlf network. I have two organizations. peers - First org - peer0.wipro.com, peer1.wipro.com, orderer0.wipro.com, second org - peer0.infosys.com, orderer1.infosys.com. I have three channels - mychannel (common to both org), wiprochannel(only wipro) and infosyschannel(only infosys)
but I can see error in peer container logs
2020-09-18 16:44:08.341 UTC [policies] EvaluateSignedData -> DEBU 588 == Evaluating *policies.ImplicitMetaPolicy Policy /Channel/Application/Readers ==
2020-09-18 16:44:08.341 UTC [policies] EvaluateSignedData -> DEBU 589 This is an implicit meta policy, it will trigger other policy evaluations, whose failures may be benign
2020-09-18 16:44:08.341 UTC [policies] EvaluateSignedData -> DEBU 58a == Evaluating *cauthdsl.policy Policy /Channel/Application/infosysMSP/Readers ==
2020-09-18 16:44:08.341 UTC [msp.identity] Verify -> DEBU 58b Verify: digest = 00000000 90 e0 d0 8e 56 90 d5 0f 15 7a 11 2f eb 39 42 7e |....V....z./.9B~|
00000010 92 4c 76 e5 7b 53 d9 b5 74 d5 6d 0f 74 67 7b ea |.Lv.{S..t.m.tg{.|
2020-09-18 16:44:08.341 UTC [msp.identity] Verify -> DEBU 58c Verify: sig = 00000000 30 45 02 21 00 b4 0a cb 81 fd 05 ce 79 e3 9f 96 |0E.!........y...|
00000010 a0 53 9c 3e c3 66 cf 56 56 e0 73 3d 90 42 83 59 |.S.>.f.VV.s=.B.Y|
00000020 16 c9 07 4c 81 02 20 2b bc fc 5b 57 51 25 01 ac |...L.. +..[WQ%..|
00000030 4f 58 a8 1b 0f 83 6f b3 5e bc c9 3c b0 6c 96 77 |OX....o.^..<.l.w|
00000040 32 e6 a4 17 4c 6a 33 |2...Lj3|
2020-09-18 16:44:08.342 UTC [policies] SignatureSetToValidIdentities -> DEBU 58d signature for identity 0 validated
2020-09-18 16:44:08.342 UTC [cauthdsl] func1 -> DEBU 58e 0xc0038264e0 gate 1600447448342192297 evaluation starts
2020-09-18 16:44:08.342 UTC [cauthdsl] func2 -> DEBU 58f 0xc0038264e0 signed by 0 principal evaluation starts (used [false])
2020-09-18 16:44:08.342 UTC [cauthdsl] func2 -> DEBU 590 0xc0038264e0 processing identity 0 - &{wiproMSP 27e87c82790285560a6629d3e139a5a58878fb6078f47cbf2d991e525a0df4c9}
2020-09-18 16:44:08.342 UTC [cauthdsl] func2 -> DEBU 591 0xc0038264e0 identity 0 does not satisfy principal: the identity is a member of a different MSP (expected infosysMSP, got wiproMSP)
2020-09-18 16:44:08.342 UTC [cauthdsl] func2 -> DEBU 592 0xc0038264e0 principal evaluation fails
2020-09-18 16:44:08.342 UTC [cauthdsl] func1 -> DEBU 593 0xc0038264e0 gate 1600447448342192297 evaluation fails
2020-09-18 16:44:08.342 UTC [policies] EvaluateSignedData -> DEBU 594 Signature set did not satisfy policy /Channel/Application/infosysMSP/Readers
2020-09-18 16:44:08.342 UTC [policies] EvaluateSignedData -> DEBU 595 == Done Evaluating *cauthdsl.policy Policy /Channel/Application/infosysMSP/Readers
2020-09-18 16:44:08.342 UTC [policies] func1 -> DEBU 596 Evaluation Failed: Only 0 policies were satisfied, but needed 1 of [ infosysMSP/Readers ]
2020-09-18 16:44:08.342 UTC [policies] EvaluateSignedData -> DEBU 597 Signature set did not satisfy policy /Channel/Application/Readers
2020-09-18 16:44:08.343 UTC [policies] EvaluateSignedData -> DEBU 598 == Done Evaluating *policies.ImplicitMetaPolicy Policy /Channel/Application/Readers
2020-09-18 16:44:08.343 UTC [gossip.comm] authenticateRemotePeer -> ERRO 599 Failed verifying signature from 15.207.20.123:7051 : implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Readers' sub-policies to be satisfied
2020-09-18 16:44:08.343 UTC [gossip.comm] Handshake -> WARN 59a Authentication failed: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Readers' sub-policies to be satisfied
2020-09-18 16:44:08.343 UTC [grpc] Infof -> DEBU 59b Channel Connectivity change to SHUTDOWN
2020-09-18 16:44:08.343 UTC [grpc] Infof -> DEBU 59c Subchannel Connectivity change to SHUTDOWN
2020-09-18 16:44:08.343 UTC [gossip.gossip] func1 -> WARN 59d Deep probe of peer0.wipro.com:7051 failed: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Readers' sub-policies to be satisfied
2020-09-18 16:44:08.343 UTC [gossip.discovery] func1 -> WARN 59e Could not connect to Endpoint: peer0.wipro.com:7051, InternalEndpoint: peer0.wipro.com:7051, PKI-ID:
yacovm (Mon, 12 Oct 2020 18:28:38 GMT):
[blabla](http://example.com)
troyronda (Wed, 28 Oct 2020 17:46:00 GMT):
Has left the channel.
rahulkundani (Thu, 31 Dec 2020 13:03:15 GMT):
Has joined the channel.
Sandyzhanghs (Sun, 10 Jan 2021 04:57:21 GMT):
Has joined the channel.
bh4rtp (Thu, 18 Feb 2021 00:35:49 GMT):
Does anyone summarize which ports should be opened by firewall to enable gossip over different hosts?
bh4rtp (Thu, 18 Feb 2021 01:48:09 GMT):
I built a fabric network on 2 VMs hosted in an old thinkpad x230. To simulate concurrent proformance, five clients submit transactions simutaeously. About after 900 seconds, it failed to disseminate private data.
```2021-02-17T17:43:02.789Z - error: [Transaction]: Error: No valid responses from any peers. Errors:
peer=peer1.example.com, status=500, message=error in simulation: failed to distribute private collection, txID 6cddc86a2eb75033381e2d12f2a1acc52cfc8f244622ab7af0ac4b7d44a99d47, channel mychannel: Failed disseminating 2 out of 8 private dissemination plans```
Is this issue caused by poor resources?
bh4rtp (Thu, 18 Feb 2021 01:48:09 GMT):
I built a fabric network on 2 VMs hosted in an old thinkpad x230. To simulate concurrent proformance, five clients submit transactions simutaeously. About after 900 seconds, it failed to disseminate private data.
```2021-02-17T17:43:02.789Z - error: [Transaction]: Error: No valid responses from any peers. Errors:
peer=peer1.org1.example.com, status=500, message=error in simulation: failed to distribute private collection, txID 6cddc86a2eb75033381e2d12f2a1acc52cfc8f244622ab7af0ac4b7d44a99d47, channel mychannel: Failed disseminating 2 out of 8 private dissemination plans```
Is this issue caused by poor resources?
bh4rtp (Thu, 18 Feb 2021 01:48:09 GMT):
I built a fabric network on 2 VMs hosted in an old thinkpad x230. To simulate concurrent proformance, five clients submit transactions simutaeously. About after 900 seconds, it failed to disseminate private data.
```2021-02-17T17:43:02.789Z - error: [Transaction]: Error: No valid responses from any peers. Errors:
peer=peer1.org1.example.com, status=500, message=error in simulation: failed to distribute private collection, txID 6cddc86a2eb75033381e2d12f2a1acc52cfc8f244622ab7af0ac4b7d44a99d47, channel mychannel: Failed disseminating 2 out of 8 private dissemination plans```
Is this issue caused by poor resources?
bh4rtp (Thu, 18 Feb 2021 01:48:09 GMT):
I built a fabric network on 2 VMs hosted in an old thinkpad x230. To simulate concurrent proformance, five clients submit transactions simutaeously. About after 900 seconds, it failed to disseminate private data.
```2021-02-17T17:43:02.789Z - error: [Transaction]: Error: No valid responses from any peers. Errors:
peer=peer1.org1.example.com, status=500, message=error in simulation: failed to distribute private collection, txID 6cddc86a2eb75033381e2d12f2a1acc52cfc8f244622ab7af0ac4b7d44a99d47, channel mychannel: Failed disseminating 2 out of 8 private dissemination plans```
Is this issue caused by shortage of resources?
bh4rtp (Thu, 04 Mar 2021 00:35:37 GMT):
this kind of issue occurred last night.
```2021-03-03 19:10:33.384 CST [gossip.gossip] SendByCriteria -> WARN 1173 Failed sending to 10.20.0.107:11051 error: timed out
2021-03-03 19:10:33.385 CST [gossip.privdata] func1 -> ERRO 1174 Failed disseminating private RWSet for TxID 01fbe93fb092ba250ca797e7fbb9632e2b59ba7e53966d72b9b8c76c3b3b60f7 , namespace mychaincode collection PrivateCollection : {"failures":{"timed out":1}} channel=salech
2021-03-03 19:10:33.385 CST [gossip.service] DistributePrivateData -> ERRO 1175 failed to distribute private collection, txID 01fbe93fb092ba250ca797e7fbb9632e2b59ba7e53966d72b9b8c76c3b3b60f7, channel mychannel: Failed disseminating 3 out of 8 private dissemination plans```
kayterina (Mon, 07 Jun 2021 15:10:48 GMT):
Has joined the channel.
AndreyP 3 (Tue, 14 Sep 2021 08:44:47 GMT):
Has joined the channel.
rjones (Wed, 23 Mar 2022 17:24:52 GMT):
rjones (Wed, 23 Mar 2022 17:24:52 GMT):
rjones (Wed, 23 Mar 2022 17:24:52 GMT):
