rjones (Fri, 08 Sep 2017 16:41:46 GMT):
kostas
rjones (Fri, 08 Sep 2017 16:42:43 GMT):
User User_1 added by rjones.
rjones (Fri, 08 Sep 2017 16:42:50 GMT):
jyellick
rjones (Fri, 08 Sep 2017 16:42:58 GMT):
jyellick
rjones (Fri, 08 Sep 2017 16:43:04 GMT):
kostas
jyellick (Fri, 08 Sep 2017 16:45:38 GMT):
Development discussions only. Please take user questions to #fabric-consensus
kostas (Fri, 08 Sep 2017 16:45:49 GMT):
@sanchezl @guoger
guoger (Fri, 08 Sep 2017 16:45:49 GMT):
Has joined the channel.
sanchezl (Fri, 08 Sep 2017 16:45:49 GMT):
Has joined the channel.
rjones (Fri, 08 Sep 2017 17:04:25 GMT):
Has left the channel.
kostas (Sat, 09 Sep 2017 18:47:15 GMT):
Do we have any idea why, say, the msp logger's debug statements show up when running orderer tests?
kostas (Sat, 09 Sep 2017 18:47:27 GMT):
```~/Go/src/github.com/hyperledger/fabric/msp (master)* $ pt flogging
mspmgrimpl.go
22: "github.com/hyperledger/fabric/common/flogging"
28:var mspLogger = flogging.MustGetLogger("msp")
identities.go
31: "github.com/hyperledger/fabric/common/flogging"
36:var mspIdentityLogger = flogging.MustGetLogger("msp/identity")
cache/cache.go
15: "github.com/hyperledger/fabric/common/flogging"
26:var mspLogger = flogging.MustGetLogger("msp")
mgmt/mgmt.go
26: "github.com/hyperledger/fabric/common/flogging"
64:var mspLogger = flogging.MustGetLogger("msp")
```
kostas (Sat, 09 Sep 2017 18:48:54 GMT):
As best as I can tell, they don't modify the logging spec, and the `flogging` package sets the default level to INFO:
kostas (Sat, 09 Sep 2017 18:48:59 GMT):
`defaultLevel = logging.INFO`
kostas (Sat, 09 Sep 2017 19:07:10 GMT):
I added a debug statement to the `flogging` package to confirm the level at which each module is initialized, and this is what I see: https://pastebin.com/xZZnGi2H
kostas (Sat, 09 Sep 2017 19:07:10 GMT):
I added an INFO-level statement to the `flogging` package to confirm the level at which each module is initialized, and this is what I see: https://pastebin.com/xZZnGi2H
kostas (Sat, 09 Sep 2017 19:07:10 GMT):
I added an INFO-level log statement to the `flogging` package to confirm the level at which each module is initialized, and this is what I see: https://pastebin.com/xZZnGi2H
jyellick (Sun, 10 Sep 2017 01:44:59 GMT):
@kostas Often in our test files, we often have a:
```
func init() {
logging.SetLevel(logging.DEBUG, "")
}
```
jyellick (Sun, 10 Sep 2017 01:46:10 GMT):
The assumption being that usually, the log output is suppressed, except in case of failure, when the debug level is desirable
kostas (Sun, 10 Sep 2017 19:06:18 GMT):
@jyellick: Ah, all of my queries were `flogging.*` which is why I was missing this, thanks.
kostas (Sun, 10 Sep 2017 19:06:22 GMT):
Not sure I get the "log output is suppressed" part however.
kostas (Sun, 10 Sep 2017 19:06:26 GMT):
Failure or not, the output does not get suppressed, right?
kostas (Sun, 10 Sep 2017 19:06:30 GMT):
(It's the reason why I've defaulted the log level to "error" in the `kafka` package tests.)
jyellick (Sun, 10 Sep 2017 19:46:39 GMT):
Unless I execute `go test -v` and the test passes, all log output is suppressed
jyellick (Sun, 10 Sep 2017 19:47:08 GMT):
For example:
```
[yellickj@jmobile orderer]$ go test ./consensus/solo/
ok github.com/hyperledger/fabric/orderer/consensus/solo 0.114s
[yellickj@jmobile orderer]$
```
kostas (Sun, 10 Sep 2017 20:05:22 GMT):
Hm. How about this then? https://pastebin.com/L9EL0MAC
jyellick (Sun, 10 Sep 2017 20:24:50 GMT):
Interesting, it seems to be the difference between:
```
go test
```
and
```
[yellickj@jmobile server]$ go test .
ok github.com/hyperledger/fabric/orderer/common/server 1.689s
[yellickj@jmobile server]$
```
jyellick (Sun, 10 Sep 2017 20:25:03 GMT):
(I always run the tests with a `.` if that is my intent
jyellick (Sun, 10 Sep 2017 20:25:16 GMT):
I had assumed that `go test` was imply an alias for `go test .`
kostas (Sun, 10 Sep 2017 20:48:24 GMT):
Yeah, not sure what's up either.
kostas (Sun, 10 Sep 2017 20:48:30 GMT):
Do you have any idea why the `msp` or `viperutil` packages do debug-level printouts though?
kostas (Sun, 10 Sep 2017 20:48:43 GMT):
As best as I can tell, the `msp` package has ` logging.SetLevel(logging.DEBUG...`) statement but it's on the `init` function of their `*_test.go`files, so it shouldn't be activated unless we run those specific tests in the `msp` package.
kostas (Sun, 10 Sep 2017 20:48:52 GMT):
(And there is no such statement on the `viperutil` package.)
kostas (Sun, 10 Sep 2017 22:16:00 GMT):
@guoger: I've spent the past couple of days working with your performance/benchmark code, which is generally excellent.
kostas (Sun, 10 Sep 2017 22:16:08 GMT):
I'm getting all sorts of failures however when doing Kafka-based tests, with deliver threads panicking, and the `fsblkstorage` returning errors (see: https://chat.hyperledger.org/channel/fabric-ledger?msg=eTcrrDE5WmY3v3jTa).
kostas (Sun, 10 Sep 2017 22:16:27 GMT):
(I should note that I don't expect this to be a disk size issue, as I'm running this from within my Vagrant box, but have modified your test so that it creates the orderer ledgers in the `/vagrant` dir, which maps to my host machine, where disk space is plenty.)
kostas (Sun, 10 Sep 2017 22:17:53 GMT):
I will give you a more detailed report tomorrow, but for now a question, in case you (or anyone else in here) has a take:
kostas (Sun, 10 Sep 2017 22:18:31 GMT):
Should we provide a `docker-compose.yml` file for bringing up a base ZK/Kafka configuration for those benchmark tests, or should we let that be the user's problem?
kostas (Sun, 10 Sep 2017 22:19:57 GMT):
Providing a Docker Compose config file would be nice, but:
kostas (Sun, 10 Sep 2017 22:20:43 GMT):
a. I am slightly skeptical of turning this test into a heavy integration test that calls into the OS to bring up a Docker composition
kostas (Sun, 10 Sep 2017 22:21:03 GMT):
b. Writing a Docker Composition that just works out of the box is slightly trickier than it sounds
kostas (Sun, 10 Sep 2017 22:21:03 GMT):
b. Writing a Docker composition that just works out of the box is slightly trickier than it sounds
kostas (Sun, 10 Sep 2017 22:22:45 GMT):
For my own local tests, I originally attempted to modify the `dc-orderer-*.yml` files under the `/bddtests` directory by just adding port mappings to the host machine.
kostas (Sun, 10 Sep 2017 22:23:12 GMT):
The thinking was that I would just target those ports on localhost, and voila, I'd be able to interact with a Kafka/ZK cluster that was brought up via Docker Compose.
kostas (Sun, 10 Sep 2017 22:23:49 GMT):
However this doesn't work, cause ZK records the addresses as reported by the Kafka brokers, and it is those addresses that get sent back to the Kafka client (i.e. our test).
kostas (Sun, 10 Sep 2017 22:24:04 GMT):
Those addresses however make sense in the Docker Compose network namespace, not the host machine.
kostas (Sun, 10 Sep 2017 22:27:05 GMT):
Long story short, I spent a ton of time trying to set up Kafka listeners (internal/external) for that purpose but failed to get it to work. (Related KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separation+of+Internal+and+External+traffic)
kostas (Sun, 10 Sep 2017 22:28:16 GMT):
So now I'm using a hacked-up Docker Compose file where all Kafka/ZK services use the host for network (`network_mode: "host"`).
kostas (Sun, 10 Sep 2017 22:29:12 GMT):
But even that doesn't completely solve the problem as you need to add an `extra_hosts`key to get it work, whose value is host-dependent.
kostas (Sun, 10 Sep 2017 22:29:12 GMT):
But even that doesn't completely solve the problem as you need to add an `extra_hosts`key to get it to work, whose value is host-dependent.
kostas (Sun, 10 Sep 2017 22:29:12 GMT):
But even that doesn't completely solve the problem as you need to add an `extra_hosts`key to get it to work, whose value is host-dependent. (I just love Docker.)
kostas (Sun, 10 Sep 2017 22:29:12 GMT):
But even that doesn't completely solve the problem as you need to add an `extra_hosts`key to get it to work, whose value is host-dependent.
kostas (Sun, 10 Sep 2017 22:29:15 GMT):
See: http://blog.yohanliyanage.com/2016/09/docker-machine-moby-name-or-service-not-known/
kostas (Sun, 10 Sep 2017 22:30:09 GMT):
So that value is, say, `"ubuntu-xenial:127.0.0.1` on my Vagrant machine and `mobdy:127.0.0.1` on my Mac.
kostas (Sun, 10 Sep 2017 22:30:49 GMT):
(And on the Mac it doesn't really work, as host networking has always been problematic for Docker as best as I can tell. I am not an expert though.)
kostas (Sun, 10 Sep 2017 22:32:08 GMT):
TL;DR -- There's definitely something up with the Kafka tests, and I'll need some of your cycles there.
kostas (Sun, 10 Sep 2017 22:32:10 GMT):
Meanwhile, if any of you has thoughts on how/whether we should make provisioning a Kafka/ZK cluster for those tests easy, let me know.
jyellick (Sun, 10 Sep 2017 23:03:18 GMT):
> As best as I can tell, the `msp` package has ` logging.SetLevel(logging.DEBUG...`) statement but it's on the `init` function of their `*_test.go`files, so it shouldn't be activated unless we run those specific tests in the `msp` package.
@kostas That `SetLogging` is global for all packages, not just the local package, we could probably refine that somehow
jyellick (Sun, 10 Sep 2017 23:29:41 GMT):
(and of course the contents of one package's `_test.go` files cannot have any affect on another's `_test.go` files (as when `go test` builds the test binary, it is one binary per package, where the binary is the code imported by that package and its test files.)
guoger (Mon, 11 Sep 2017 03:55:39 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-consensus-dev?msg=Gv3z265ucuee7mowX) @kostas I'm looking at the log you sent to me. Is the failure constant or flaky?
guoger (Mon, 11 Sep 2017 03:59:33 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-consensus-dev?msg=ThuY4ze3G95GMTQyK) @kostas ah, I've tried this briefly when I was working on the benchmark and ran into same issue. but you've been going farther now. I agree that we should spin up a kafka cluster for each test, but the work is non-trivial, so I only left a TODO for the time being.
guoger (Mon, 11 Sep 2017 05:46:00 GMT):
@kostas I looked at log2.txt file you sent to me. I came across the same issue when I worked on it and I talked to @manish-sethi . Unfortunately the test was flaky for me and I couldn't reproduce it under single channel.
manish-sethi (Mon, 11 Sep 2017 05:46:00 GMT):
Has joined the channel.
guoger (Mon, 11 Sep 2017 05:46:35 GMT):
here's the log I sent to him https://pastebin.com/9p3fMxYr , seem to be identical to log2.txt.
guoger (Mon, 11 Sep 2017 05:47:26 GMT):
we could take this to #fabric-ledger if necessary
kostas (Mon, 11 Sep 2017 15:41:50 GMT):
@guoger: I pushed out a changeset to bring us all on the same page: https://gerrit.hyperledger.org/r/c/13325/
kostas (Mon, 11 Sep 2017 15:43:03 GMT):
I get the failure in a single channel (single-everything really) as soon as I bump the total transaction count to 10K.
kostas (Mon, 11 Sep 2017 15:43:34 GMT):
Could you please spend some cycles with Manish --if he's available-- to get to the bottom of what's wrong here?
guoger (Mon, 11 Sep 2017 15:49:14 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-consensus-dev?msg=Ybawi3xexPXkSAvkc) @kostas OK, will do
kostas (Mon, 11 Sep 2017 16:22:57 GMT):
@jyellick: RE: https://chat.hyperledger.org/channel/fabric-consensus-dev?msg=7KDHqi6vvJ5NzCm5P
kostas (Mon, 11 Sep 2017 16:23:06 GMT):
Let me preface this by saying that:
kostas (Mon, 11 Sep 2017 16:23:15 GMT):
1. I am dead sure I'm missing something obvious, and
kostas (Mon, 11 Sep 2017 16:23:15 GMT):
1. I am dead sure I'm missing something obvious
kostas (Mon, 11 Sep 2017 16:23:33 GMT):
2. I am however still not sure what that thing is
kostas (Mon, 11 Sep 2017 16:23:40 GMT):
Let me also give a concrete example:
kostas (Mon, 11 Sep 2017 16:23:50 GMT):
`cd orderer/common/server; go test`
kostas (Mon, 11 Sep 2017 16:24:14 GMT):
You'll see the usual verbose suspects there: `msp`, `viperutil`
kostas (Mon, 11 Sep 2017 16:24:42 GMT):
Now, as best as I can tell, we're doing that global setting of the logger only in test files:
kostas (Mon, 11 Sep 2017 16:24:45 GMT):
https://pastebin.com/wsg16EtM
kostas (Mon, 11 Sep 2017 16:25:33 GMT):
Ah, damn.
kostas (Mon, 11 Sep 2017 16:25:51 GMT):
Do you think it's the statement in `server_test.go` that's causing this?
jyellick (Mon, 11 Sep 2017 16:26:06 GMT):
Yes
jyellick (Mon, 11 Sep 2017 16:29:03 GMT):
I believe if instead of `logging.SetLevel(logging.DEBUG, "")` we did `logging.SetLevel(logging.DEBUG, "orderer/server/main")`` things would be good
jyellick (Mon, 11 Sep 2017 16:29:03 GMT):
I believe if instead of `logging.SetLevel(logging.DEBUG, "")` we did `logging.SetLevel(logging.DEBUG, "orderer/server/main")` things would be good
jyellick (Mon, 11 Sep 2017 16:29:03 GMT):
I believe if instead of `logging.SetLevel(logging.DEBUG, "")` we did `logging.SetLevel(logging.DEBUG, "orderer/server/main")` things would be good Edit: Damn, that did not work in my local test
jyellick (Mon, 11 Sep 2017 16:29:41 GMT):
I'd also point out that many of our logging package names are wrong, this should be `orderer/common/server` not `orderer/server/main` I would think.
jyellick (Mon, 11 Sep 2017 16:29:48 GMT):
@kostas ^
kostas (Mon, 11 Sep 2017 16:33:56 GMT):
@jyellick: That was it, you are right.
kostas (Mon, 11 Sep 2017 16:34:07 GMT):
`logging.SetLevel(logging.DEBUG, "orderer/server/main")` did the trick.
jyellick (Mon, 11 Sep 2017 16:34:16 GMT):
Locally, I found a second issue
kostas (Mon, 11 Sep 2017 16:34:17 GMT):
This was pissing me off so much during the weekend.
kostas (Mon, 11 Sep 2017 16:34:32 GMT):
I always had to scroll down to find the output I was interested in.
kostas (Mon, 11 Sep 2017 16:35:05 GMT):
What's the second issue?
jyellick (Mon, 11 Sep 2017 16:35:23 GMT):
```
[yellickj@jmobile server]$ git diff .
diff --git a/orderer/common/server/main_test.go b/orderer/common/server/main_test.go
index c2fd52a15..7931f3586 100644
--- a/orderer/common/server/main_test.go
+++ b/orderer/common/server/main_test.go
@@ -29,6 +29,7 @@ import (
)
func TestInitializeLoggingLevel(t *testing.T) {
+ t.Skip()
initializeLoggingLevel(
&config.TopLevel{
General: config.General{LogLevel: "debug"},
diff --git a/orderer/common/server/server_test.go b/orderer/common/server/server_test.go
index 000a851ed..26700a809 100644
--- a/orderer/common/server/server_test.go
+++ b/orderer/common/server/server_test.go
@@ -26,7 +26,7 @@ import (
)
func init() {
- logging.SetLevel(logging.DEBUG, "")
+ logging.SetLevel(logging.DEBUG, "orderer/server/main")
}
func TestBroadcastNoPanic(t *testing.T) {
```
jyellick (Mon, 11 Sep 2017 16:35:27 GMT):
This was my required fix
jyellick (Mon, 11 Sep 2017 16:35:27 GMT):
Without skipping the `TestInitializeLoggingLevel` I was still seeing verbose output
jyellick (Mon, 11 Sep 2017 16:35:53 GMT):
Because the test for initializing the logging additionally sets logging to DEBUG and never switches it back
jyellick (Mon, 11 Sep 2017 16:36:03 GMT):
Not sure why you are not seeing that in your test?
kostas (Mon, 11 Sep 2017 16:54:14 GMT):
Ah, because I switched to running in the meantime: `go test -run TestOrdererBenchmarkSolo`
kostas (Mon, 11 Sep 2017 16:56:38 GMT):
I'm going to push those changes in a performance-related changeset I have out there.
kostas (Mon, 11 Sep 2017 16:56:45 GMT):
Thanks for helping me get to the bottom of this.
jyellick (Mon, 11 Sep 2017 17:13:03 GMT):
For anyone who's interested, especially @kostas https://jira.hyperledger.org/browse/FAB-6108
kostas (Mon, 11 Sep 2017 17:30:10 GMT):
Thanks for writing this up. The pattern that I'm following is:
kostas (Mon, 11 Sep 2017 17:52:26 GMT):
Define a `pkgLogID` constant (see: https://github.com/hyperledger/fabric/blob/release/orderer/kafka/consenter.go#L18) and then use the `init` functions on the main+test files to initialize the logger properly. For example:
kostas (Mon, 11 Sep 2017 17:52:33 GMT):
https://github.com/hyperledger/fabric/blob/release/orderer/kafka/consenter.go#L23
kostas (Mon, 11 Sep 2017 17:52:50 GMT):
https://github.com/hyperledger/fabric/blob/release/orderer/kafka/consenter_test.go#L159
kostas (Mon, 11 Sep 2017 17:53:15 GMT):
I'll try to convert the rest of the `orderer` package to this pattern.
kostas (Mon, 11 Sep 2017 17:53:51 GMT):
It still feels like a hacky solution though, and I wish we, as Fabric in general, had a better way of handling logging.
jyellick (Mon, 11 Sep 2017 17:55:03 GMT):
+1 to this pattern, I'd also like to see this fixed up in `fabric/common` since it gets into the orderer logs
kostas (Mon, 11 Sep 2017 18:52:44 GMT):
Quick question for you folks:
kostas (Mon, 11 Sep 2017 18:53:16 GMT):
The pattern for logger modules had always been: `path-relative-to-fabric-dir/package-name`
kostas (Mon, 11 Sep 2017 18:53:25 GMT):
So: `orderer/common/server`
kostas (Mon, 11 Sep 2017 18:53:42 GMT):
That'll give us some nasty stuttering for the ledger implementations:
kostas (Mon, 11 Sep 2017 18:53:58 GMT):
`orderer/ledger/file/fileledger`
kostas (Mon, 11 Sep 2017 18:53:58 GMT):
`orderer/ledger/fileledger`
kostas (Mon, 11 Sep 2017 18:54:01 GMT):
Are we OK with this?
jyellick (Mon, 11 Sep 2017 20:17:06 GMT):
I am not crazy about it, but I would opt for consistency over stuttering
sanchezl (Mon, 11 Sep 2017 20:31:45 GMT):
There are some known problems with running leveldb (and hence our ledger) on a VirtualBox shared drive. The results are seemingly random "I/O Errors". (Example
Message: 500000 Message Size: 10KB Channels: 10 Orderer(kafka): 5 | Broadcast Clients: 500 Write tps: 393.6 tx/s Elapsed Time: 1270.25s | Deliver clients: 0 Read tps: 0.0 blk/s Elapsed Time: 1270.25s
PASS
ok github.com/hyperledger/fabric/orderer/common/server 1325.424s
```
sanchezl (Mon, 11 Sep 2017 20:35:49 GMT):
There are some known problems with running leveldb (and hence our ledger) on a VirtualBox shared drive. The results are seemingly random "I/O Errors". (Example thread where others have encountered this problem, and the "solution" was to stop using VirtualBox: https://github.com/Level/levelup/issues/222). So, using a host drive in Vagrant is not going to work.
I can run locally on macOS with Kafka running on Docker. Here are the results from one such run:
```
Message: 500000 Message Size: 10KB Channels: 10 Orderer(kafka): 5 | Broadcast Clients: 500 Write tps: 393.6 tx/s Elapsed Time: 1270.25s | Deliver clients: 0 Read tps: 0.0 blk/s Elapsed Time: 1270.25s
PASS
ok github.com/hyperledger/fabric/orderer/common/server 1325.424s
```
kostas (Mon, 11 Sep 2017 20:39:39 GMT):
Awesome, thank you Luis.
kostas (Mon, 11 Sep 2017 20:40:50 GMT):
https://gerrit.hyperledger.org/r/c/13345/ BTW is the changeset for log package name consistency
jyellick (Mon, 11 Sep 2017 20:42:02 GMT):
I am still blown away by the fact that changing the Gerrit UI breaks linking
kostas (Mon, 11 Sep 2017 20:42:30 GMT):
Oh wait, my links don't work for the old UI?
jyellick (Mon, 11 Sep 2017 20:42:37 GMT):
They do not
kostas (Mon, 11 Sep 2017 20:42:39 GMT):
Nice.
kostas (Mon, 11 Sep 2017 20:42:42 GMT):
Sorry about that.
jyellick (Mon, 11 Sep 2017 20:42:55 GMT):
No problem, easy enough to fix the link, just feels like something Gerrit should do for us
jyellick (Mon, 11 Sep 2017 20:47:57 GMT):
One minor comment on 13345, feel free to fix or not, +2-ed
jyellick (Mon, 11 Sep 2017 20:47:57 GMT):
@kostas One minor comment on 13345, feel free to fix or not, +2-ed
kostas (Mon, 11 Sep 2017 20:49:47 GMT):
Oh no, it's a good catch. Will fix.
kostas (Wed, 13 Sep 2017 02:35:48 GMT):
https://chat.hyperledger.org/channel/fabric-consensus?msg=fpbScFhQKtW6SHcZR
kostas (Wed, 13 Sep 2017 02:36:21 GMT):
@guoger: As best as I can tell I should be done with all of the pending reviews, but if I'm missing something, please LMK.
kostas (Wed, 13 Sep 2017 02:36:21 GMT):
@guoger: As best as I can tell I should be done with all of the pending reviews, but if I'm missing something, please LMK. (Sorry for that delay.)
guoger (Wed, 13 Sep 2017 02:37:33 GMT):
Hi I just landed in LA this afternoon. I will start addressing them today
kostas (Wed, 13 Sep 2017 02:38:18 GMT):
(No rush.)
kostas (Wed, 13 Sep 2017 02:44:50 GMT):
Are we good with moving this to 1.2?https://jira.hyperledger.org/browse/FAB-5542
kostas (Wed, 13 Sep 2017 02:44:50 GMT):
Are we good with moving this to 1.2? https://jira.hyperledger.org/browse/FAB-5542
guoger (Wed, 13 Sep 2017 02:46:52 GMT):
LGTM
guoger (Wed, 13 Sep 2017 02:47:04 GMT):
done
jyellick (Wed, 13 Sep 2017 05:55:39 GMT):
@guoger I notice that you're using the same FAB-XXXX for related CRs (which is why you'll see I accidentlaly closed, and then re-opened it).
The approach I've been taking is to create a sub-task for each CR, and linking to that JIRA item in the CR. This way it's more obvious how much work is still remaining for an issue. Then, once all of the sub-tasks close, I close the parent issue. Any opinions on this @kostas ?
(I also realize that in this case specifically you were operating on a sub-task already, so this would have been hard to do).
kostas (Wed, 13 Sep 2017 06:03:35 GMT):
I agree that every changeset should map to a different JIRA. Got an email notification that 5284 was merged 10d ago and almost had a heart attack.
guoger (Wed, 13 Sep 2017 06:04:30 GMT):
ah, ok, I'll try to use this approach in the future. For this specific sub-task, I'll make sure to update jira myself accordingly.
jyellick (Wed, 13 Sep 2017 06:05:35 GMT):
@guoger I wouldn't at all be opposed to promoting this sub-task to a proper issue, and simply linking to 5284
jyellick (Wed, 13 Sep 2017 06:06:26 GMT):
Perhaps a bit late for this one, but in the future, if an sub-task cannot be completed in one CR, then I think promoting it to its own improvement, and placing a block link against the original parent is the way to go.
jyellick (Wed, 13 Sep 2017 06:06:26 GMT):
Perhaps a bit late for this one, but in the future, if an sub-task cannot be completed in one CR, then I think promoting it to its own improvement, and placing a "blocks" link against the original parent is the way to go.
guoger (Wed, 13 Sep 2017 06:07:11 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-consensus-dev?msg=rcaXmnw6CrD5ZhNHS) @jyellick Sounds good, will do in the future.
guoger (Wed, 13 Sep 2017 08:45:33 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-consensus-dev?msg=RxMKcng7wtjxQhY5j) @sanchezl I vaguely remember I ran into the same issue without using virtualbox. (I ran tests on mac directly when I worked on benchmark tests) but it was definitely flaky for me. Will let you know if I ever manage to reproduce the issue. But good to know the problem with leveldb, thanks for the info!
guoger (Wed, 13 Sep 2017 21:03:37 GMT):
@kostas just fyi for https://gerrit.hyperledger.org/r/#/c/12989/, there were more arguments supplied than required for several log format. Just updated the patch, waiting for CI to be green.
lehors (Thu, 14 Sep 2017 07:11:13 GMT):
Has joined the channel.
guoger (Fri, 15 Sep 2017 22:54:26 GMT):
@kostas I post some comments in https://gerrit.hyperledger.org/r/#/c/12991/ to address your comment.
guoger (Tue, 19 Sep 2017 00:36:49 GMT):
anyone knows why almost every proto.pb.go files change when I ONLY changed `kafka.proto` and run `make protos`?
guoger (Tue, 19 Sep 2017 00:36:49 GMT):
anyone knows why almost every *.pb.go file changes when I ONLY changed `kafka.proto` and run `make protos`?
jyellick (Tue, 19 Sep 2017 01:27:35 GMT):
Odds are someone checked in protos generated at the wrong version
jyellick (Tue, 19 Sep 2017 01:27:53 GMT):
The first proto file alphabetically in a directory will get modified regardless with a new comment
jyellick (Tue, 19 Sep 2017 01:28:05 GMT):
But protos changing in other directories means someone mis-compiled the protos
jyellick (Tue, 19 Sep 2017 01:31:02 GMT):
Actually
jyellick (Tue, 19 Sep 2017 01:31:10 GMT):
Looks like this is a go 1.9 issue
jyellick (Tue, 19 Sep 2017 01:40:31 GMT):
@guoger After executing `make clean` and `make protos` I see no diff in the protos against master
guoger (Tue, 19 Sep 2017 02:10:32 GMT):
maybe it's just my local dirty env then...
guoger (Tue, 19 Sep 2017 02:14:04 GMT):
@jyellick I'm adding `Resubmission` capability and have a question: it is turned on simply by defining it in yaml, regardless of the bool value. In another word, if I want a v1.1 orderer running in compatibility mode, I should NOT define it in the yaml.
jyellick (Tue, 19 Sep 2017 02:31:41 GMT):
@guoger Correct, if nothing is set, you should get v1.0 compat mode. If the capability is turned on, only then will you get the v1.1 behavior.
guoger (Tue, 19 Sep 2017 02:33:32 GMT):
I see. IMHO, this appears to be not very intuitive tho. At first glance, I would expect `Resubmission: false` actually disables the functionality, which however is not the case.
jyellick (Tue, 19 Sep 2017 02:34:50 GMT):
Hm? The CR referenced is: https://gerrit.hyperledger.org/r/#/c/13263/ ?
jyellick (Tue, 19 Sep 2017 02:35:17 GMT):
I would suggest you piggy-back onto the Orderer_v1.1_BugFixes capability
jyellick (Tue, 19 Sep 2017 02:35:28 GMT):
Because I believe the resubmission should be default behavior
jyellick (Tue, 19 Sep 2017 02:36:45 GMT):
In general we should avoid propagating capability strings if at all possible. If there is behavior which we would have not made configurable for v1.0, then it should fall under the v1.1 capability (and not something explicit)
guoger (Tue, 19 Sep 2017 02:46:13 GMT):
hmm.. I see, but this semantic still appears to be obscure to me. Let's say we add a capability X which is NOT to address a defect, but a feature which can be switched on/off. If it is to be added into v1.x_capability without being explicitly defined, then it couldn't be combined with another feature in v1.y version?
guoger (Tue, 19 Sep 2017 02:46:13 GMT):
hmm.. I see, but this semantic still appears to be obscure to me. Let's say we add a capability X which is NOT to address a defect, but a feature which can be switched on/off. If it is to be added into v1.x_capability without being explicitly defined, then it couldn't be decoupled with another feature in v1.x?
guoger (Tue, 19 Sep 2017 02:48:22 GMT):
Also, per design doc of compatibility, explicit capabilities also appear to be idiomatic pattern?
guoger (Tue, 19 Sep 2017 02:50:55 GMT):
After all, when thinking of the work `Capability`, one would instinctively associate it with "being able to do something, e.g. resubmit a tx, use a new msp, etc", instead of "fixing the issues in previous version"
guoger (Tue, 19 Sep 2017 02:50:55 GMT):
After all, when thinking of the work `Capability`, one would probably associate it with "being able to do something, e.g. resubmit a tx, use a new msp, etc", instead of "fixing the issues in previous version"
jyellick (Tue, 19 Sep 2017 02:51:37 GMT):
Ah, so, the design doc could use some work. In the interest of maintainability, we're trying to stick to 'versions as capability' for the time being.
jyellick (Tue, 19 Sep 2017 02:51:47 GMT):
Until we have an explicit requirement to make this otherwise
jyellick (Tue, 19 Sep 2017 02:52:14 GMT):
The real danger being that we end up with 10 peer capabilities, 10 orderer capabiltiies, and 100 test matrix combinations to try to make sure that everything works as expected.
jyellick (Tue, 19 Sep 2017 02:52:46 GMT):
It can certainly be argued that we do not test all permutations of our config options for example, but in general, unless we have a compelling reason to do otherwise, minimizing the size of this matrix is desirable
guoger (Tue, 19 Sep 2017 02:54:19 GMT):
OK. This leads to another question, how do we plan to do compatibility test?
jyellick (Tue, 19 Sep 2017 02:56:17 GMT):
@jeffgarratt Is working on some bdd upgrade paths, and of course the test team will be performing assorted scenarios
jeffgarratt (Tue, 19 Sep 2017 02:56:17 GMT):
Has joined the channel.
jyellick (Tue, 19 Sep 2017 02:56:57 GMT):
The limit of what we can do in UT with respect to upgrade is pretty limited obviously
guoger (Tue, 19 Sep 2017 03:02:49 GMT):
got it, thx!
guoger (Tue, 19 Sep 2017 20:48:56 GMT):
To address your comment in https://gerrit.hyperledger.org/r/#/c/12991/:
If you are talking about the code, sorry I'm not following.. could you elaborate?
If you are talking about the comment, I agree they probably need some rework, but I want to make sure we are on the same page first. see my reply inline:
> 1. In the compatibility case, the message you just processed is not identified by `originalOffset`.
In compatibility mode, `lastOriginalOffsetProcessed` is always zero, as should be `originalOffset` passed into the function
> 2. In the case of a normal message that gets committed on first pass, this value is not associated with that message in any way either. (See line 600.)
For messages (normal or config) get committed on the first pass, this value should be the same as current `lastOriginalOffsetProcessed`.
I try to make `commit*Msg` functions agnostic about the new offset, so it blindly update `lastOriginalOffsetProcessed` with the `originalOffset` value passed in. And it's up to the caller to decide the correct value to pass in. To echo your previous comment:
> As a caller of this function, I can't immediately tell what to set this parameter to. Is it the original offset of the message that I'm trying to commit? It is it the LastOriginalOffset of the chain when calling this function?
I think you're right that arg name could be chosen more wisely, but I don't understand the 'chain' in `newChainOriginalOffset`? maybe `newOffset` *plus* a comment?
guoger (Tue, 19 Sep 2017 20:48:56 GMT):
@kostas To address your comment in https://gerrit.hyperledger.org/r/#/c/12991/:
If you are talking about the code, sorry I'm not following.. could you elaborate?
If you are talking about the comment, I agree they probably need some rework, but I want to make sure we are on the same page first. see my reply inline:
> 1. In the compatibility case, the message you just processed is not identified by `originalOffset`.
In compatibility mode, `lastOriginalOffsetProcessed` is always zero, as should be `originalOffset` passed into the function
> 2. In the case of a normal message that gets committed on first pass, this value is not associated with that message in any way either. (See line 600.)
For messages (normal or config) get committed on the first pass, this value should be the same as current `lastOriginalOffsetProcessed`.
I try to make `commit*Msg` functions agnostic about the new offset, so it blindly update `lastOriginalOffsetProcessed` with the `originalOffset` value passed in. And it's up to the caller to decide the correct value to pass in. To echo your previous comment:
> As a caller of this function, I can't immediately tell what to set this parameter to. Is it the original offset of the message that I'm trying to commit? It is it the LastOriginalOffset of the chain when calling this function?
I think you're right that arg name could be chosen more wisely, but I don't understand the 'chain' in `newChainOriginalOffset`? maybe `newOffset` *plus* a comment?
kostas (Tue, 19 Sep 2017 21:26:35 GMT):
@guoger: Which comment of all?
guoger (Tue, 19 Sep 2017 21:27:04 GMT):
> I do not think this is accurate.
1. In the compatibility case, the message you just processed is not identified by `originalOffset`.
2. In the case of a normal message that gets committed on first pass, this value is not associated with that message in any way either. (See line 600.)
guoger (Tue, 19 Sep 2017 21:27:04 GMT):
> I do not think this is accurate.
> 1. In the compatibility case, the message you just processed is not identified by `originalOffset`.
> 2. In the case of a normal message that gets committed on first pass, this value is not associated with that message in any way either. (See line 600.)
kostas (Tue, 19 Sep 2017 21:27:16 GMT):
Ah right.
guoger (Tue, 19 Sep 2017 21:27:31 GMT):
https://gerrit.hyperledger.org/r/#/c/12991/8/orderer/consensus/kafka/chain.go@448
guoger (Tue, 19 Sep 2017 21:27:41 GMT):
Just realized you could send a link to specific comment..
kostas (Tue, 19 Sep 2017 21:27:52 GMT):
> In compatibility mode, `lastOriginalOffsetProcessed` is always zero, as should be `originalOffset` passed into the function
kostas (Tue, 19 Sep 2017 21:28:11 GMT):
I do not dispute that. (And I do not doubt the validity of the code, I've inspected it.)
kostas (Tue, 19 Sep 2017 21:28:21 GMT):
What I claim is wrong is this:
kostas (Tue, 19 Sep 2017 21:28:32 GMT):
> The message we have just processed (originalOffset) is encapsulated
kostas (Tue, 19 Sep 2017 21:28:54 GMT):
This implies to me that the just processed message is associated with `originalOffset` in a way.
kostas (Tue, 19 Sep 2017 21:29:13 GMT):
It is not, for either of those two cases that I mentioned.
guoger (Tue, 19 Sep 2017 21:29:28 GMT):
I see, I've reworked the comment a bit: https://pastebin.com/rqpGNb4X
kostas (Tue, 19 Sep 2017 21:29:32 GMT):
I will repeat that I get and agree with the code. I do not agree (or misinterpret?) the comment.
kostas (Tue, 19 Sep 2017 21:30:08 GMT):
> I think you're right that arg name could be chosen more wisely, but I don't understand the 'chain' in `newChainOriginalOffset`? maybe `newOffset`*plus* a comment?
kostas (Tue, 19 Sep 2017 21:32:19 GMT):
Variable names are highly subjective and I'll leave the final choice to you. To answer your question, I add `chain` in that name, because what you're doing in this function is to essentially set the chain's `lastOriginalOffsetProcessed` field. (Perhaps `newChainOriginalOffsetProcessed` would be more expressive, but I'm afraid the var name is becoming ridiculously long.)
kostas (Tue, 19 Sep 2017 21:33:06 GMT):
As I wrote in my original comment, a comment will work just as well: https://gerrit.hyperledger.org/r/#/c/12991/8/orderer/consensus/kafka/chain.go@431
kostas (Tue, 19 Sep 2017 21:33:29 GMT):
(Reading your Pastebin link now.)
kostas (Tue, 19 Sep 2017 21:34:38 GMT):
Lines 2-7 on that Pastebin look great BTW.
guoger (Tue, 19 Sep 2017 21:34:59 GMT):
I'm trying to get this message crossed:
> I try to make `commit*Msg` functions agnostic about the new offset, so it blindly update `lastOriginalOffsetProcessed` with the `originalOffset` value passed in. And it's up to the caller to decide the correct value to pass in
guoger (Tue, 19 Sep 2017 21:34:59 GMT):
I'm trying to get this message crossed:
> I try to make `commit*Msg` functions agnostic about the new offset, so it blindly update `lastOriginalOffsetProcessed` with the `newOffset` value passed in. And it's up to the caller to decide the correct value to pass in
kostas (Tue, 19 Sep 2017 21:35:39 GMT):
I think you're absolutely right to do that.
kostas (Tue, 19 Sep 2017 21:35:56 GMT):
(And this is what I get as a reader when reading your updated comments.)
kostas (Tue, 19 Sep 2017 21:35:56 GMT):
(And this is what I get as a reader when going over your updated comments.)
guoger (Tue, 19 Sep 2017 21:37:50 GMT):
I'm about to update the CR to address all other comments as well, unless you have some immediate feedbacks
kostas (Tue, 19 Sep 2017 21:38:39 GMT):
I'm giving it a quick scan and everything LGTM so far.
kostas (Tue, 19 Sep 2017 21:38:39 GMT):
I'm giving it a quick look and everything LGTM so far.
guoger (Tue, 19 Sep 2017 21:51:05 GMT):
To address your comment regarding UT. Resubmission logic is mostly covered by `TestResubmission` and the coverage is 87.7% for now.
guoger (Tue, 19 Sep 2017 21:51:10 GMT):
except for some error cases
guoger (Tue, 19 Sep 2017 21:51:33 GMT):
however, this is definitely not enough and I'm trying to see how to add integration tests
guoger (Tue, 19 Sep 2017 21:51:39 GMT):
cc @kostas
guoger (Tue, 19 Sep 2017 21:56:14 GMT):
by the way, I got this question that if orderer could be switched from `solo` to `kafka`. I told them this is explicitly not-supported. However, I'm curious if we plan to support switching consensus algorithm in the future. Not sure if there's a real use case though. There's a Chinese company (Tencent, which is actually a tech giant here) that's doing this in their blockchain platform, to improve throughput. I don't know much details tho, but I assume doing a high-throughput consensus for most of time and occasionally switch to log-throughput one for exceptions.
kostas (Tue, 19 Sep 2017 22:07:19 GMT):
This is indeed explicitly not supported. We decide on the roadmap collaboratively but I am highly skeptical of the benefits of this feature, relative to the engineering effort that will need to be put in place to support it. There are lower (and far more useful) fruits to grab first.
Colonel_HLE (Wed, 20 Sep 2017 12:19:48 GMT):
Has joined the channel.
guoger (Thu, 21 Sep 2017 05:24:42 GMT):
@jyellick replied https://gerrit.hyperledger.org/r/#/c/12991/
jyellick (Thu, 21 Sep 2017 05:33:55 GMT):
@guoger Back at you
guoger (Thu, 21 Sep 2017 05:51:34 GMT):
@jyellick updated the patch. Regarding `Ordered()` api, I'm not sure what you mean by
> I wonder if this API should not simply be redesigned
Reverting it to `Ordered(msg *cb.Envelope) ([][]*cb.Envelope, bool)`?
jyellick (Thu, 21 Sep 2017 05:58:24 GMT):
Ah, no, I mean the `blockcutter` API
jyellick (Thu, 21 Sep 2017 05:58:50 GMT):
Blockcutter used to do a lot more than it does now, so it tried to hide what was happening under the covers so that the caller didn't have to care.
jyellick (Thu, 21 Sep 2017 05:59:24 GMT):
Really, it takes `*cb.Envelope`s which it should not
jyellick (Thu, 21 Sep 2017 05:59:37 GMT):
It should take byte slices to prevent remarshaling envelopes
jyellick (Thu, 21 Sep 2017 05:59:45 GMT):
As should the writeblock API
jyellick (Thu, 21 Sep 2017 06:08:29 GMT):
But beyond that, I'm wondering if instead it shouldn't simply be turned into something much different.
```
type BatchBuilder interface {
// CanAdd returns whether a given message will fit into the current batch
CanAdd(msg []byte) bool
// Add adds the message to the batch, and returns whether the batch is full or not
Add(msg []byte) bool
// Batch returns the batch of transactions
Batch() [][]byte
}
```
Or whatever primitives it makes sense to build the resubmission logic from. So, you might end up with logic that looks like:
```
if ! batchBulider.CanAdd(msg) {
// Commit the current batchBuilder.Batch() and allocate a new one
}
if batchBuilder.Add(msg) {
// Commit batchBuilder.Batch() and allocate a new one
}
```
Maybe this isn't the exact API that makes sense (or great names etc.)
jyellick (Thu, 21 Sep 2017 06:09:42 GMT):
But essentially, the `blockcutter` API has always been pretty cludgy and due for a rewrite, so if that would make life easier on the resubmission, I'd vote we do it.
guoger (Thu, 21 Sep 2017 06:16:00 GMT):
I agree the api design could use some rework, but I'm not sure how it makes resubmission logic simpler? resubmission actually happens before we even consider committing the message, however `blockCutter` determines how we commit it
jyellick (Fri, 22 Sep 2017 05:23:21 GMT):
@guoger My thought was that the commit logic contains a fair bit of flow control around how to set `lastOriginalOffsetPersisted` based on the output of `blockcutter`, which I thought might be simplified by improving that API
jyellick (Fri, 22 Sep 2017 05:23:46 GMT):
But, if you'd rather tackle the API later, that's fine, just thought it might simplify the CR
jyellick (Fri, 22 Sep 2017 05:23:46 GMT):
But, if you'd rather tackle the API later (or leave it be), that's fine, just thought it might simplify the CR
guoger (Fri, 22 Sep 2017 14:41:47 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-consensus-dev?msg=gnwXbReoDnQQ9kC8Y) @jyellick ah, I see your point now. Agreed, but I guess I'll tackle that after this CR, since refactoring blockcutter api probably deserves more thoughts. Thanks for the feedback!
guoger (Mon, 25 Sep 2017 03:52:35 GMT):
@kostas is there a jira already to track the work of blocking `Order`/`Configure` until consenter catches up?
guoger (Mon, 25 Sep 2017 03:52:35 GMT):
@kostas is there a jira already to track the work of blocking `Order`/ `Configure` until consenter catches up?
kostas (Mon, 25 Sep 2017 10:55:57 GMT):
@guoger If I’m not mistaken, there isn’t. Feel free to go for it.
guoger (Mon, 25 Sep 2017 15:42:30 GMT):
@kostas I just realized that I filed one a while ago :P https://jira.hyperledger.org/browse/FAB-5969
kostas (Mon, 25 Sep 2017 15:42:46 GMT):
Ah excellent.
guoger (Mon, 25 Sep 2017 15:44:26 GMT):
the question is, should `order`/ `configure` return error in this case?
guoger (Mon, 25 Sep 2017 15:46:46 GMT):
I guess we should simply block at broadcast (so ingress msg will queue in grpc)
kostas (Mon, 25 Sep 2017 15:47:31 GMT):
Yes, that was my initial thought as well.
kostas (Wed, 27 Sep 2017 10:29:43 GMT):
@tsariounov: Following up on our conversation yesterday, related to developing a BFT orderer, these pointers will get you started:
tsariounov (Wed, 27 Sep 2017 10:29:43 GMT):
Has joined the channel.
kostas (Wed, 27 Sep 2017 10:29:56 GMT):
A consensus plugin needs to implement the `Consenter` and `Chain` interfaces defined here:
https://github.com/hyperledger/fabric/blob/master/orderer/consensus/consensus.go
kostas (Wed, 27 Sep 2017 10:29:56 GMT):
A consensus plugin needs to implement the `Consenter` and `Chain` interfaces defined here: https://github.com/hyperledger/fabric/blob/master/orderer/consensus/consensus.go
kostas (Wed, 27 Sep 2017 10:30:09 GMT):
There are two consensus plugins currently:
https://github.com/hyperledger/fabric/tree/master/orderer/consensus/solo
https://github.com/hyperledger/fabric/tree/master/orderer/consensus/kafka
kostas (Wed, 27 Sep 2017 10:30:17 GMT):
The entire orderer code can be found here:
https://github.com/hyperledger/fabric/tree/master/orderer
kostas (Wed, 27 Sep 2017 10:30:17 GMT):
The entire orderer code can be found here: https://github.com/hyperledger/fabric/tree/master/orderer
kostas (Wed, 27 Sep 2017 10:30:27 GMT):
PBFT paper:
https://scholar.google.com/scholar?q=practical+byzantine+fault+tolerance+and+proactive+recovery
kostas (Wed, 27 Sep 2017 10:30:27 GMT):
PBFT paper: https://scholar.google.com/scholar?q=practical+byzantine+fault+tolerance+and+proactive+recovery
kostas (Wed, 27 Sep 2017 10:30:38 GMT):
Delta between SBFT and PBFT: https://jira.hyperledger.org/browse/FAB-378
binhn (Wed, 27 Sep 2017 14:28:22 GMT):
Has joined the channel.
tsariounov (Wed, 27 Sep 2017 15:25:57 GMT):
Awesome, thanks @kostas
guoger (Thu, 28 Sep 2017 04:47:20 GMT):
thank @kostas for sharing this in public channel. It would nice if more and more offline discussion can be concluded and documented for posterity
rjones (Thu, 28 Sep 2017 18:28:41 GMT):
Has joined the channel.
rjones (Thu, 28 Sep 2017 18:28:57 GMT):
Room name changed to: fabric-orderer-dev by rjones
rjones (Thu, 28 Sep 2017 18:29:08 GMT):
Development discussions only. Please take user questions to #fabric-orderer
kostas (Thu, 28 Sep 2017 19:52:11 GMT):
Marginally useful but I'll take it: https://github.com/rakyll/gotest
Ashish (Fri, 29 Sep 2017 05:14:55 GMT):
Has joined the channel.
guoger (Fri, 29 Sep 2017 17:51:12 GMT):
https://gerrit.hyperledger.org/r/#/c/13781/ adds a UT for solo consenter, pls take a look, thx!
jyellick (Mon, 02 Oct 2017 01:18:51 GMT):
There is apparently a CI failure here: https://jenkins.hyperledger.org/job/fabric-verify-z/13105/console
```
11:55:55 unit-tests_1 | 2017-10-01 11:55:55.394 UTC [orderer/kafka] Halt -> CRIT 001 [channel: 0x80a393a0}.channel] Halting of chain requested
11:55:55 unit-tests_1 | 2017-10-01 11:55:55.395 UTC [orderer/kafka] processMessagesToBlocks -> CRIT 002 [channel: 0x80a393a0}.channel] Unable to unmarshal consumed message = proto: can't skip unknown wire type 7 for orderer.KafkaMessage
11:55:55 unit-tests_1 | 2017-10-01 11:55:55.405 UTC [orderer/kafka] processMessagesToBlocks -> ERRO 003 [channel: 0x80a393a0}.channel] Error during consumption:
rjones (Mon, 02 Oct 2017 04:27:14 GMT):
Has left the channel.
cca88 (Mon, 02 Oct 2017 06:39:25 GMT):
Has joined the channel.
yacovm (Mon, 02 Oct 2017 07:07:49 GMT):
Has joined the channel.
kostas (Mon, 02 Oct 2017 20:44:42 GMT):
^^ I've created a JIRA for this, thanks.
kostas (Mon, 02 Oct 2017 20:45:18 GMT):
Going back to the discussion on https://jira.hyperledger.org/browse/FAB-6080
kostas (Mon, 02 Oct 2017 20:45:18 GMT):
Going back to the discussion on: https://jira.hyperledger.org/browse/FAB-6080
kostas (Mon, 02 Oct 2017 20:46:10 GMT):
Here's what I'm thinking.
kostas (Mon, 02 Oct 2017 20:46:57 GMT):
If we assume that 1.0.0 binaries are active there, we have a situation where they parse channel creation requests where the `mod_policy` for `/Channel` is not set.
kostas (Mon, 02 Oct 2017 20:47:18 GMT):
Any binary starting from 1.0.1 rejects these requests, as you've noted.
kostas (Mon, 02 Oct 2017 20:48:38 GMT):
@jyellick: I guess the punchline is: what do you gain by gating this behind a capability flag?
kostas (Mon, 02 Oct 2017 20:48:38 GMT):
@jyellick: I guess my question would be: what do we gain by gating this behind a capability flag?
kostas (Mon, 02 Oct 2017 20:51:59 GMT):
It is not really the avoidance of forks, since we can have these in mixed networks today already (with 1.0.0 nodes coexisting with 1.0.1 or 1.0.2 coexisting).
kostas (Mon, 02 Oct 2017 21:03:17 GMT):
I guess this is how I interpret the approach in FAB-6080:
kostas (Mon, 02 Oct 2017 21:04:29 GMT):
Now that we're getting capabilities and we're putting out a guide on how to upgrade the network, etc. let's gate behind the v1.1 key all the features that would break the network otherwise.
kostas (Mon, 02 Oct 2017 21:04:54 GMT):
(Just a sec.)
kostas (Mon, 02 Oct 2017 21:04:58 GMT):
(One more statement.)
kostas (Mon, 02 Oct 2017 21:05:30 GMT):
For _this particular feature_, we are actually pretending that the delta between 1.0.0 nodes and 1.0.1 does not exist.
kostas (Mon, 02 Oct 2017 21:05:55 GMT):
Otherwise the cynic approach would be: you guys can have networks that introduce forks already.
kostas (Mon, 02 Oct 2017 21:05:55 GMT):
Otherwise the cynical approach to this would be: you guys can have networks that introduce forks already.
kostas (Mon, 02 Oct 2017 21:06:34 GMT):
These are more or less my thoughts on FAB-6080, and why it gave me pause when we discussed this originally.
kostas (Mon, 02 Oct 2017 21:06:47 GMT):
So: I don't disagree with the approach. But do I have the context right?
jyellick (Mon, 02 Oct 2017 21:10:07 GMT):
> For _this particular feature_, we are actually pretending that the delta between 1.0.0 nodes and 1.0.1 does not exist.
Essentially yes. It could be argued that we _should_ go back and modify v1.0.3 to use the old behavior, that breaking it in v1.0.1 was a mistake, but I think this would cause us more problems than it would solve.
kostas (Mon, 02 Oct 2017 21:10:50 GMT):
Perfect, same page then.
jyellick (Mon, 02 Oct 2017 21:15:31 GMT):
I would also note that for 6080, the problem is slightly worse than the v1.0.0 to v1.0.1. In the latter case, so long as the v1.0.1 version of configtxgen is used, or the v1.0.1 orderer is targeted, there is no fork. In the case of 6080, all creation transactions, regardless of configtxgen version and regardless of version at point of ingress, results in a fork.
kostas (Mon, 02 Oct 2017 21:16:38 GMT):
Why?
jyellick (Mon, 02 Oct 2017 21:16:59 GMT):
Can you be more specific?
kostas (Mon, 02 Oct 2017 21:17:24 GMT):
Yes. Why is there a difference between the 1.0.0-1.0.1 case the FAB-6080 case?
kostas (Mon, 02 Oct 2017 21:17:24 GMT):
Yes. Why is there a difference between the 1.0.0-1.0.1 case and the FAB-6080 case?
kostas (Mon, 02 Oct 2017 21:18:42 GMT):
Assume that you target the 1.1 orderer in the FAB-6080 case, to keep things consistent with your 1.0.0-1.0.1 case example.
kostas (Mon, 02 Oct 2017 21:19:40 GMT):
You have a fork in that case, but do not if you were to target the 1.0.1 orderer in a mixed 1.0.0-1.0.1 network. And now my original question hopefully makes more sense: Why?
jyellick (Mon, 02 Oct 2017 21:23:29 GMT):
For the existing bug
badconfigtx -> OSN1.0.1 Rejects _Okay_
badconfigtx -> OSN1.0.0 Accepts -> Kafka -> OSN1.0.1 Rejects *Fork*
goodconfigtx -> OSN1.0.1 Accepts -> Kafka -> OSN1.0.0 Accepts _Okay_
goodconfigtx -> OSN1.0.0 Accepts -> Kafka -> OSN1.0.1 Accepts _Okay_
For the 6080 bug
configtx -> OSN1.1.0 Accepts -> Kafka -> OSN1.0.1 Rejects *Fork*
configtx -> OSN1.0.0 Accepts -> Kafka -> OSN1.1.0 Rejects *Fork*
The key here is that the each OSN version generates the genesis config different. And, if an OSN's computation of the genesis config does not match what it sees on the channel, it invalidates and rejects it (assuming that something about the ordering system channel changed and caused the invalidation)
jyellick (Mon, 02 Oct 2017 21:23:29 GMT):
For the existing bug
badconfigtx -> OSN1.0.1 Rejects _Okay_
badconfigtx -> OSN1.0.0 Accepts -> Kafka -> OSN1.0.1 Rejects *Fork*
goodconfigtx -> OSN1.0.1 Accepts -> Kafka -> OSN1.0.0 Accepts _Okay_
goodconfigtx -> OSN1.0.0 Accepts -> Kafka -> OSN1.0.1 Accepts _Okay_
For the 6080 bug
configtx -> OSN1.1.0 Accepts -> Kafka -> OSN1.0.1 Rejects *Fork*
configtx -> OSN1.0.0 Accepts -> Kafka -> OSN1.1.0 Rejects *Fork*
The key here is that the each OSN version generates the genesis config differently. And, if an OSN's computation of the genesis config does not match what it sees on the ordering system channel, it invalidates and rejects it (assuming that something about the ordering system channel changed and caused the invalidation)
jyellick (Mon, 02 Oct 2017 21:23:29 GMT):
For the existing bug
badconfigtx -> OSN1.0.1 Rejects _Okay_
badconfigtx -> OSN1.0.0 Accepts -> Kafka -> OSN1.0.1 Rejects *Fork*
goodconfigtx -> OSN1.0.1 Accepts -> Kafka -> OSN1.0.0 Accepts _Okay_
goodconfigtx -> OSN1.0.0 Accepts -> Kafka -> OSN1.0.1 Accepts _Okay_
For the 6080 bug (without capabilities dependency)
configtx -> OSN1.1.0 Accepts -> Kafka -> OSN1.0.1 Rejects *Fork*
configtx -> OSN1.0.0 Accepts -> Kafka -> OSN1.1.0 Rejects *Fork*
The key here is that the each OSN version generates the genesis config differently. And, if an OSN's computation of the genesis config does not match what it sees on the ordering system channel, it invalidates and rejects it (assuming that something about the ordering system channel changed and caused the invalidation)
kostas (Mon, 02 Oct 2017 21:27:56 GMT):
Can you point me to the changeset of the existing bug?
jyellick (Mon, 02 Oct 2017 21:29:03 GMT):
https://gerrit.hyperledger.org/r/#/c/11645/
kostas (Tue, 03 Oct 2017 15:08:04 GMT):
All the cases for the existing bug make sense to me, the ones for the 6080 bug however do not -- yet.
kostas (Tue, 03 Oct 2017 15:08:24 GMT):
> configtx -> OSN1.1.0 Accepts -> Kafka -> OSN1.0.1 Rejects *Fork
What kind of config update transaction would cause this?
kostas (Tue, 03 Oct 2017 15:09:30 GMT):
I assume that since we're talking about a tx that 1.0.1 rejects, it leaves the `mod_policy` empty.
kostas (Tue, 03 Oct 2017 15:11:41 GMT):
And this is the scenario where the 1.1.0 binary has been configured to accept such transaction but add the `mod_policy` itself. Correct?
kostas (Tue, 03 Oct 2017 15:11:41 GMT):
And this is the scenario where the 1.1.0 binary has been configured to accept such a transaction but add the `mod_policy` itself. Correct?
jyellick (Tue, 03 Oct 2017 15:39:22 GMT):
Ah, so, this has nothing to do with checking the validity of the `mod_policy`
jyellick (Tue, 03 Oct 2017 15:43:07 GMT):
Channel creation occurs roughly as follows:
1. Creation tx received
2. Template config generated
3. Creation tx applied to template config and validated to generate genesis config
4. Genesis config ordered
5. Prior to commit, each orderer performs 2/3, and verifies that the output of (3) matches what was received in (4). This is because (2) is derived from the current ordering system channel config, which, if it changed while the request was in flight, the creation may no longer be valid.
6. Commits
So, the 6080 bug is that we need to change the output of (2). Which in turn causes (5) to fail for any OSN who does not follow the same rules for (2).
jyellick (Tue, 03 Oct 2017 15:44:38 GMT):
So yes, although 6080 happens to change how `mod_policy` is set during (2), the fact that it is `mod_policy` is coincidental. It could have been that we accidentally created a consortiums group for the new channel when we should not have. The key is that (2) will now be different depending on v1.1.x vs v1.0.x.
kostas (Tue, 03 Oct 2017 15:46:05 GMT):
Understood. Reviewing CR 13263 so this comes handy.
jyellick (Tue, 03 Oct 2017 16:29:12 GMT):
A note, I was incorrect:
> configtx -> OSN1.1.0 Accepts -> Kafka -> OSN1.0.1 Rejects *Fork*
> configtx -> OSN1.0.0 Accepts -> Kafka -> OSN1.1.0 Rejects *Fork*
In the second case, there is no fork. Only in the first.
jyellick (Tue, 03 Oct 2017 16:29:12 GMT):
~A note, I was incorrect:~
> configtx -> OSN1.1.0 Accepts -> Kafka -> OSN1.0.1 Rejects *Fork*
> configtx -> OSN1.0.0 Accepts -> Kafka -> OSN1.1.0 Rejects *Fork*
~In the second case, there is no fork. Only in the first.~
Actually, I think I was right the first time around. Sorry for the waffling.
jyellick (Tue, 03 Oct 2017 16:31:14 GMT):
I was considering that in the v1.1 case, the orderers do not revalidate transactions before committing them. However, this is not true, because we follow the validate and commit path for messages which do not have a config seq set in Kafka, so there would in fact be the fork.
kostas (Tue, 03 Oct 2017 16:35:34 GMT):
In the review for CR13263 (https://gerrit.hyperledger.org/r/c/13263#message-df002fed_b68be80c) you write:
kostas (Tue, 03 Oct 2017 16:35:44 GMT):
> Conversely, a v1.0 orderer receives the creation request, and a v1.1 orderer will successfully commit it.
jyellick (Tue, 03 Oct 2017 16:35:55 GMT):
Yes, this prompted me to make the correction here
jyellick (Tue, 03 Oct 2017 16:35:58 GMT):
Then reverse
jyellick (Tue, 03 Oct 2017 16:36:02 GMT):
So that CR comment is incorrect
kostas (Tue, 03 Oct 2017 16:36:06 GMT):
Ah, good.
jyellick (Tue, 03 Oct 2017 16:40:06 GMT):
Perhaps it is too extraneous to bother with, but I will note that if the logic for (2) is changed in v1.2, there will be no non-determinism risk with v1.1 orderers, as the validation is done only at ingress, not at commit in this case.
jyellick (Tue, 03 Oct 2017 20:26:36 GMT):
@guoger @sanchezl I just experienced this Kafka failure:
```
? github.com/hyperledger/fabric/orderer/consensus [no test files]
2017-10-03 16:25:06.849 EDT [orderer/consensus/kafka] Halt -> CRIT 001 [channel: 0xc4200d84b0.channel] Halting of chain requested
2017-10-03 16:25:06.849 EDT [orderer/consensus/kafka] processMessagesToBlocks -> CRIT 002 [channel: 0xc4200d84b0.channel] Unable to unmarshal consumed message = proto: can't skip unknown wire type 7 for orderer.KafkaMessage
2017-10-03 16:25:06.850 EDT [orderer/consensus/kafka] Halt -> CRIT 003 [channel: 0xc4200d84b0.channel] Halting of chain requested
2017-10-03 16:25:06.850 EDT [orderer/consensus/kafka] processMessagesToBlocks -> CRIT 004 [channel: 0xc4200d84b0.channel] Unable to unmarshal consumed message = proto: can't skip unknown wire type 7 for orderer.KafkaMessage
2017-10-03 16:25:06.850 EDT [orderer/consensus/kafka] processMessagesToBlocks -> ERRO 005 [channel: 0xc4200d84b0.channel] Error during consumption:
jyellick (Tue, 03 Oct 2017 20:26:36 GMT):
@guoger @sanchezl I just experienced this Kafka failure:
```
? github.com/hyperledger/fabric/orderer/consensus [no test files]
2017-10-03 16:25:06.849 EDT [orderer/consensus/kafka] Halt -> CRIT 001 [channel: 0xc4200d84b0.channel] Halting of chain requested
2017-10-03 16:25:06.849 EDT [orderer/consensus/kafka] processMessagesToBlocks -> CRIT 002 [channel: 0xc4200d84b0.channel] Unable to unmarshal consumed message = proto: can't skip unknown wire type 7 for orderer.KafkaMessage
2017-10-03 16:25:06.850 EDT [orderer/consensus/kafka] Halt -> CRIT 003 [channel: 0xc4200d84b0.channel] Halting of chain requested
2017-10-03 16:25:06.850 EDT [orderer/consensus/kafka] processMessagesToBlocks -> CRIT 004 [channel: 0xc4200d84b0.channel] Unable to unmarshal consumed message = proto: can't skip unknown wire type 7 for orderer.KafkaMessage
2017-10-03 16:25:06.850 EDT [orderer/consensus/kafka] processMessagesToBlocks -> ERRO 005 [channel: 0xc4200d84b0.channel] Error during consumption:
AlekNS (Wed, 04 Oct 2017 05:14:08 GMT):
Has joined the channel.
carlosfaria (Wed, 04 Oct 2017 12:50:34 GMT):
Has joined the channel.
sanchezl (Wed, 04 Oct 2017 13:14:30 GMT):
I'll investigate the CI failure today.
jyellick (Fri, 06 Oct 2017 17:55:54 GMT):
@sanchezl Any update on the above?
jyellick (Fri, 06 Oct 2017 17:55:54 GMT):
@sanchezl Any update on the above? I am still seeing this with reasonable consistency when I run the orderer tests locally
sanchezl (Fri, 06 Oct 2017 18:04:23 GMT):
@jyellick not yet
sanchezl (Fri, 06 Oct 2017 18:51:15 GMT):
@jyellick , do you also see this similar error?
```11:55:55 unit-tests_1 | 2017-10-01 11:55:55.395 UTC [orderer/kafka] processMessagesToBlocks -> CRIT 002 [channel: 0x80a393a0}.channel] Unable to unmarshal consumed message = proto: can't skip unknown wire type 7 for orderer.KafkaMessage
11:55:55 unit-tests_1 | 2017-10-01 11:55:55.405 UTC [orderer/kafka] processMessagesToBlocks -> ERRO 003 [channel: 0x80a393a0}.channel] Error during consumption:
jyellick (Fri, 06 Oct 2017 18:52:04 GMT):
@sanchezl No, I have not seen tha tone, just the one I posted above
jyellick (Fri, 06 Oct 2017 18:52:04 GMT):
@sanchezl No, I have not seen that one, just the one I posted above
yacovm (Fri, 06 Oct 2017 21:25:11 GMT):
Here is another one in case you need: https://jenkins.hyperledger.org/job/fabric-verify-z/13333/console
yacovm (Fri, 06 Oct 2017 21:25:11 GMT):
Here is another 2 in case you need: https://jenkins.hyperledger.org/job/fabric-verify-z/13333/console and https://jenkins.hyperledger.org/job/fabric-verify-z/13334/console
jyellick (Fri, 06 Oct 2017 21:26:11 GMT):
Thanks, would you mind looking into that one as well @sanchezl?
sanchezl (Fri, 06 Oct 2017 21:26:53 GMT):
Will do..
yacovm (Tue, 10 Oct 2017 07:44:55 GMT):
Perhaps it would be a good idea to disable the test @jyellick / @kostas / @sanchezl and work it offline?
yacovm (Tue, 10 Oct 2017 07:45:07 GMT):
it fails quite often
kostas (Tue, 10 Oct 2017 08:25:41 GMT):
Yes, if the issue persists, this is the way to go.
kostas (Tue, 10 Oct 2017 08:35:43 GMT):
@guoger: Rebase https://gerrit.hyperledger.org/r/c/12991/ when you get a moment so we can get it in?
guoger (Tue, 10 Oct 2017 08:38:02 GMT):
@sanchezl I wonder which failure is https://gerrit.hyperledger.org/r/#/c/14325 targeting? I'm just back from vacation and I want to see which failure is not yet covered
guoger (Tue, 10 Oct 2017 08:38:14 GMT):
@kostas ok, will do
guoger (Tue, 10 Oct 2017 12:36:35 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=AbCv4XrtiJ45zmWxf) @kostas done
sanchezl (Tue, 10 Oct 2017 13:19:35 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=YicKuW5xCCiMrM9Sr) @guoger This covers multiple failures that have been occurring intermittently when a chain is haltedduring CI. I have patched some of them in the past (by adding checks for nil, etc) , but I think I finally got to the true issue with this change set.
guoger (Tue, 10 Oct 2017 15:00:27 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=jiipJJr4yFaqefKxb) @sanchezl ah, gotcha. thx!
fz (Wed, 11 Oct 2017 17:07:18 GMT):
Has joined the channel.
kostas (Wed, 11 Oct 2017 19:51:39 GMT):
What’s becoming painfully obvious to me from the discussions in #fabric-orderer is that no amount of warnings, caveat emptors, etc. will give inexperienced users pause or prevent them from taking a shortcut w/r/t Kafka.
kostas (Wed, 11 Oct 2017 19:51:50 GMT):
I wish I had a good and useful conclusion here.
kostas (Wed, 11 Oct 2017 19:56:59 GMT):
And to be clear: if there is blame to be assigned here, I am not suggesting that it’s on the users.
jyellick (Wed, 11 Oct 2017 19:57:01 GMT):
Creating this channel has turned out to be an excellent idea, as the signal to noise ratio here is much nicer. I'm not sure what to do about #fabric-orderer but continue to do our best to plow through it.
jyellick (Wed, 11 Oct 2017 19:58:33 GMT):
As you say, no amount of "Please study Kafka first" seems to be taken seriously, and unless it is, I'm not sure what can be done.
yoheiueda (Thu, 12 Oct 2017 04:02:04 GMT):
Has joined the channel.
guoger (Thu, 12 Oct 2017 09:21:36 GMT):
how do we reverify e2e-x86 tests?
guoger (Thu, 12 Oct 2017 09:46:10 GMT):
fyi, I just updated https://gerrit.hyperledger.org/r/c/13963/ for review
kostas (Thu, 12 Oct 2017 12:33:42 GMT):
> how do we reverify e2e-x86 tests?
`rebuild e2e`
kostas (Thu, 12 Oct 2017 12:33:42 GMT):
> how do we reverify e2e-x86 tests?
`rebuild-e2e` (edited)
kostas (Thu, 12 Oct 2017 12:34:06 GMT):
There was an issue with E2E tests that _should_ be resolved now, see: https://chat.hyperledger.org/channel/fabric-pr-review?msg=vHyRH7gxXb5S6o8s5
jyellick (Thu, 12 Oct 2017 13:40:48 GMT):
> `rebuild e2e`
Is it not `rebuild-e2e` ?
kostas (Thu, 12 Oct 2017 13:41:09 GMT):
Sigh, it is.
kostas (Thu, 12 Oct 2017 13:41:09 GMT):
Sigh, of course it is.
kostas (Wed, 18 Oct 2017 00:42:22 GMT):
Slightly off-topic but since it's handy for development:
kostas (Wed, 18 Oct 2017 00:42:27 GMT):
If you find yourself reading code on GitHub repos often, the SourceGraph extension for Chrome is fantastic: https://about.sourcegraph.com/
kostas (Wed, 18 Oct 2017 14:54:20 GMT):
I believe this is good to merge: https://gerrit.hyperledger.org/r/c/12991/
jyellick (Wed, 18 Oct 2017 15:08:46 GMT):
I will take a look shortly
jyellick (Wed, 18 Oct 2017 16:08:21 GMT):
@kostas @guoger Added a comment on what looks like a bug to me (though perhaps it's not)
guoger (Wed, 18 Oct 2017 16:19:03 GMT):
@jyellick so if `pending || len(batches) == 2`, we need to persist *current* `chain.lastOriginalOffsetProcessed` first, and then update `chain.lastOriginalOffsetProcessed` with `newOffset`, and persist that into the second block (if any)
guoger (Wed, 18 Oct 2017 16:19:10 GMT):
does this address your question?
jyellick (Wed, 18 Oct 2017 16:19:21 GMT):
Not quite
jyellick (Wed, 18 Oct 2017 16:19:54 GMT):
As best as I can tell, `lastOriginalOffsetProcessed` is only set in those commit paths, and only when a block is cut.
jyellick (Wed, 18 Oct 2017 16:20:09 GMT):
It seems like we're losing offsets that have been processed any time a batch is not cut.
jyellick (Wed, 18 Oct 2017 16:20:44 GMT):
Since we set `lastOriginalOffsetProcessed` based only on the offset of the transaction which caused the block to commit.
guoger (Wed, 18 Oct 2017 16:27:31 GMT):
ah, I see your point, it seems to be a good catch! I'll fix it after dinner
guoger (Wed, 18 Oct 2017 21:38:35 GMT):
Anybody knows why I'm getting
```
ld: warning: PIE disabled. Absolute addressing (perhaps -mdynamic-no-pic) not allowed in code signed PIE, but used in type..eqfunc.[106]string from /var/folders/6z/9kwhngss6f9372k4fbnxks5r0000gp/T/go-link-396036579/go.o. To fix this warning, don't compile with -mdynamic-no-pic or link with -Wl,-no_pie
```
guoger (Wed, 18 Oct 2017 21:38:42 GMT):
while running unit tests
guoger (Wed, 18 Oct 2017 21:38:42 GMT):
while running unit tests on OSX
jyellick (Wed, 18 Oct 2017 22:05:04 GMT):
I've not seen it. I'd suggest a `make clean-all` if you haven't already?
guoger (Thu, 19 Oct 2017 08:12:14 GMT):
I tried... and still
guoger (Thu, 19 Oct 2017 08:13:16 GMT):
which go version are using? 1.9?
jyellick (Thu, 19 Oct 2017 13:27:06 GMT):
Yes
jyellick (Thu, 19 Oct 2017 13:27:19 GMT):
Though on Linux, not OS X
kostas (Thu, 19 Oct 2017 21:43:08 GMT):
Jay given that our CI is like rolling a dice, on this one "reverify-behave" would have sufficed: https://gerrit.hyperledger.org/r/c/12991/
kostas (Thu, 19 Oct 2017 21:43:08 GMT):
Jay given that our CI is like rolling a dice, on [this CR](https://gerrit.hyperledger.org/r/c/12991/) a "reverify-behave" would have sufficed.
kostas (Thu, 19 Oct 2017 21:43:23 GMT):
Just a heads up so that you don't have to deal with additional failures.
kostas (Thu, 19 Oct 2017 21:43:48 GMT):
(The two-staged CI job that failed doesn't actually count against you so you can ignore that one.)
kostas (Thu, 19 Oct 2017 21:44:19 GMT):
> Anybody knows why I'm getting
I'm running unit tests on my Mac on the latest master to see if I'll get this, will update you.
jyellick (Fri, 20 Oct 2017 13:57:23 GMT):
> on this CR a "reverify-behave"
I believe you want 'rebuild-behave', running `'reverify-behave' will likely trigger all the CI jobs again.
kostas (Fri, 20 Oct 2017 14:22:02 GMT):
I do this mistake. Every. Single. Time.
kostas (Fri, 20 Oct 2017 19:49:17 GMT):
@guoger: Unit tests running on this Mac w/o issues. Mac OS 10.12.6.
guoger (Sun, 22 Oct 2017 14:03:12 GMT):
hmm... thanks! that's weird..
guoger (Sun, 22 Oct 2017 14:04:46 GMT):
are you running `make unit-test` or `go test ./...`?
kostas (Mon, 23 Oct 2017 17:22:32 GMT):
@guoger: Both.
guoger (Tue, 24 Oct 2017 15:53:58 GMT):
I saw your comments in https://gerrit.hyperledger.org/r/#/c/13963/ , appreciated! indeed I missed some cases there. RW lock sounds good to me, I'll update the patch tomorrow.
jyellick (Tue, 24 Oct 2017 15:54:37 GMT):
Thanks @guoger !
guoger (Tue, 24 Oct 2017 16:08:24 GMT):
@sanchezl I see you are working on https://jira.hyperledger.org/browse/FAB-1223 , if you need any help with it, I'd love to join the effort, just let me know! thx
guoger (Wed, 25 Oct 2017 16:53:15 GMT):
I've posted a draft update of https://gerrit.hyperledger.org/r/#/c/13963, pls take a look. Probably you could pay more attention to following aspects:
- I think it's not necessary to use a `RWMutex` as we are essentially dealing with a semaphore here. So I pivoted @jyellick 's idea to use a channel
- only resubmitted config messages block the broadcast, not normal messages.
- a new field `LastResubmittedConfigOffset` is added to both `chainImpl` and `kafka.proto`, so that it gets persisted in the block. When we load the metadata from last block, we compare it with `LastOriginalOffsetProcessed` to determine if we need to block ingress messages or not
- we only unblock broadcast when `regularMessage.OriginalOffset == chain.lastResubmittedConfigOffset` && `regularMessage.ConfigSeq == seq` so that we've received the very last resubmitted message AND it doesn't need to be revalidated and resubmitted anymore
cc @jyellick @sanchezl
guoger (Wed, 25 Oct 2017 16:53:15 GMT):
I've posted a draft update of https://gerrit.hyperledger.org/r/#/c/13963, pls take a look. Probably you could pay more attention to following aspects:
- I think it's not necessary to use a `RWMutex` as we are essentially dealing with a semaphore here. So I pivoted @jyellick 's idea to use a channel
- only resubmitted config messages block the broadcast, not normal messages.
- a new field `LastResubmittedConfigOffset` is added to both `chainImpl` and `kafka.proto`, so that it gets persisted in the block. When we load the metadata from last block, we compare it with `LastOriginalOffsetProcessed` to determine if we need to block ingress messages or not
- we only unblock broadcast when `regularMessage.OriginalOffset == chain.lastResubmittedConfigOffset` && `regularMessage.ConfigSeq == seq` so that we've received the very last resubmitted message AND it doesn't need to be revalidated and resubmitted anymore
cc @jyellick @sanchezl @kostas
guoger (Wed, 25 Oct 2017 16:53:15 GMT):
I've posted a draft update of https://gerrit.hyperledger.org/r/#/c/13963, pls take a look. Probably you could pay more attention to following aspects:
- I think it's not necessary to use a `RWMutex` as we are essentially dealing with a semaphore here. So I pivoted @jyellick 's idea to use a channel. But, do we risk closing a closed channel here?
- only resubmitted config messages block the broadcast, not normal messages.
- a new field `LastResubmittedConfigOffset` is added to both `chainImpl` and `kafka.proto`, so that it gets persisted in the block. When we load the metadata from last block, we compare it with `LastOriginalOffsetProcessed` to determine if we need to block ingress messages or not
- we only unblock broadcast when `regularMessage.OriginalOffset == chain.lastResubmittedConfigOffset` && `regularMessage.ConfigSeq == seq` so that we've received the very last resubmitted message AND it doesn't need to be revalidated and resubmitted anymore
cc @jyellick @sanchezl @kostas
guoger (Wed, 25 Oct 2017 16:53:28 GMT):
I haven't updated the tests yet, will do later
guoger (Wed, 25 Oct 2017 16:53:28 GMT):
I haven't updated the tests yet, will do later. Just wanna push out this implementation change for review
guoger (Thu, 26 Oct 2017 03:42:23 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=42CSkWePsWPQbA6sA) Just FYI, this is caused by 01c50e, where we introduced `plugin` package and it has a bug in go1.9 on OSX, see https://github.com/golang/go/issues/21776, and there's a fix: https://github.com/golang/go/commit/88a1e85c706a7917f97be9c612334f0c85e96cf3, which is targeting go1.10
guoger (Thu, 26 Oct 2017 03:42:23 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=42CSkWePsWPQbA6sA) Just FYI, this is caused by commit 01c50e, where we introduced `plugin` package and it has a bug in go1.9 on OSX, see https://github.com/golang/go/issues/21776, and there's a fix: https://github.com/golang/go/commit/88a1e85c706a7917f97be9c612334f0c85e96cf3, which is targeting go1.10
guoger (Thu, 26 Oct 2017 03:42:23 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=42CSkWePsWPQbA6sA) Just FYI, this is caused by commit `01c50e`, where we introduced `plugin` package and it has a bug in go1.9 on OSX, see https://github.com/golang/go/issues/21776, and there's a fix: https://github.com/golang/go/commit/88a1e85c706a7917f97be9c612334f0c85e96cf3, which is targeting go1.10
guoger (Thu, 26 Oct 2017 03:44:46 GMT):
I'll try with go1.9.2
guoger (Thu, 26 Oct 2017 05:15:29 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=K4YkxMZQ5QaXZ7LYN) >people using plugin will probably just have to live with the warning :(
but I'm really curious why @kostas doesn't have such problem? which OSX version are you using?
guoger (Thu, 26 Oct 2017 05:15:29 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=K4YkxMZQ5QaXZ7LYN) > people using plugin will probably just have to live with the warning :(
but I'm really curious why @kostas doesn't have such problem? which OSX version are you using?
guoger (Thu, 26 Oct 2017 05:15:29 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=K4YkxMZQ5QaXZ7LYN) > people using plugin will probably just have to live with the warning :( https://github.com/golang/go/issues/21776#issuecomment-336521276
but I'm really curious why @kostas doesn't have such problem? which OSX version are you using?
guoger (Thu, 26 Oct 2017 05:15:29 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=K4YkxMZQ5QaXZ7LYN) > people using plugin will probably just have to live with the warning :( https://github.com/golang/go/issues/21776#issuecomment-336521276
but I'm really curious why @kostas doesn't have such problem.... i guess only difference is the OSX version...
guoger (Thu, 26 Oct 2017 07:43:09 GMT):
just updated tests in https://gerrit.hyperledger.org/r/#/c/13963, pls take a look, thx!
kostas (Thu, 26 Oct 2017 10:53:59 GMT):
> but I'm really curious why @kostas doesn't have such problem.... i guess only difference is the OSX version...
Ah, there is a misunderstanding. Thought you were suggesting that you cannot _run_ the tests. I also get the warning but since it's just a warning, I ignore it.
kostas (Thu, 26 Oct 2017 13:11:36 GMT):
> just updated tests in https://gerrit.hyperledger.org/r/#/c/13963, pls take a look, thx!
@guoger: Thanks! Folks, let's please get this reviewed today. (Will be looking at it myself in the PM.) Want to get it merged before the end of the week so that it makes the cut for the preview.
guoger (Thu, 26 Oct 2017 13:56:05 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=hHQx6Qr2TeennM7q8) @kostas oh, I see... now we know where that warning is coming from :) it's annoying...
guoger (Thu, 26 Oct 2017 13:56:05 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=hHQx6Qr2TeennM7q8) @kostas oh, I see... now we know where that warning is coming from :) it's annoying...especially when i run `go test ./...`
Baha-sk (Thu, 26 Oct 2017 18:41:41 GMT):
Has joined the channel.
Baha-sk (Thu, 26 Oct 2017 18:42:20 GMT):
hi, can I ask ask a Fabric newbie question about orderers and Fabric?
jyellick (Thu, 26 Oct 2017 18:42:48 GMT):
@Baha-sk Please use #fabric-orderer for that purpose
jyellick (Thu, 26 Oct 2017 18:42:48 GMT):
@Baha-sk Please use #fabric-orderer for that purpose (this channel is for development related discussions only)
Baha-sk (Thu, 26 Oct 2017 18:43:12 GMT):
ok thanks @jyellick
guoger (Fri, 27 Oct 2017 14:50:34 GMT):
@kostas Just saw your comment in FAB-5969 JIRA
> The Description is different to the solution proposed in the CR, right?
Why do you think they are different?
kostas (Fri, 27 Oct 2017 15:01:25 GMT):
> A valid configure message TX config will advance ConfigSeq when committed. In this case, while a TX config is in flight, all messages received between receipt and commit of TX config need to be re-validated. In the case of Kafka-base OSN, this introduces overhead of re-submitting those messages to Kafka (see FAB-5720 for the reason). Therefore we should block ingress messages while there are TX config is in flight.
My interpretation of this quote is as follows: if an OSN receives a config message, it should block ingress messages.
kostas (Fri, 27 Oct 2017 15:01:25 GMT):
> A valid configure message TX config will advance ConfigSeq when committed. In this case, while a TX config is in flight, all messages received between receipt and commit of TX config need to be re-validated. In the case of Kafka-base OSN, this introduces overhead of re-submitting those messages to Kafka (see FAB-5720 for the reason). Therefore we should block ingress messages while there are TX config is in flight.
My dry interpretation of this quote is as follows: if an OSN receives a config message, it should block ingress messages.
guoger (Fri, 27 Oct 2017 16:03:18 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=cRp6pNyG9aE5p6Yc7) @kostas Ah...you're right, I guess I was being careless. The idea in that jira may seem to be much simpler, but the problem I see there is that we don't have a mechanism to block all orderers *synchronously* when one of them receives a config tx via broadcast. So we could compensate by only blocking if a config tx needs to be re-submitted. Effectively we are eliminating partial overhead, but not all of it. Let me know if this makes sense to you.
kostas (Fri, 27 Oct 2017 16:07:24 GMT):
Yes, I understand what you're saying. And one could also argue that the title of that JIRA issue (or the FAB-5720 reference) should have made it clear that we're talking about the reprocessing path for configs, not the normal one. (Hence my "dry" comment on the interpretation.)
guoger (Fri, 27 Oct 2017 16:13:03 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=SoEwzWaf6HFgGMQH8) @kostas yep, and I was actually thinking of that simpler/naive solution when I created the jira, and didn't get to update the description when I pivot the idea while implementing it. Doing it now
guoger (Fri, 27 Oct 2017 16:43:10 GMT):
is there a code freeze next week?
kostas (Fri, 27 Oct 2017 19:10:13 GMT):
As best as I can tell, it won't be a code freeze. Or if there's one, it's going to be small enough in order to cut the preview release, see: https://chat.hyperledger.org/channel/fabric-release?msg=sqAiLFvB9gi3v7Fou
guoger (Sun, 29 Oct 2017 15:54:59 GMT):
@jyellick I haven't got the jira number you mentioned before :) take your time, no need to rush tho
jyellick (Mon, 30 Oct 2017 05:24:44 GMT):
Thanks, I was discussing this a bit on Friday with @kostas directly, wanted to let it percolate a bit this weekend, will try to write something up for you soon
guoger (Mon, 30 Oct 2017 08:30:27 GMT):
I talked about this problem with @jyellick earlier today:
> setting orderer `v1.1` capability to `true` doesn't take effect on master branch, verified by running e2e_cli example.
All I did is to use https://hastebin.com/dusuwopeso.coffeescript as `configtx.yaml`
I think the reason is that we are using an old version of spf13/viper, which uses dots as delimiter for nested keys, therefore key name `V1.1` is actually invalid there. Newer version of viper takes that into account, and will inspect if there are keys with dots: https://github.com/spf13/viper/blob/master/viper.go?utf8=✓#L858-L862
guoger (Mon, 30 Oct 2017 08:31:05 GMT):
I think we should consider upgrading `viper` package vendored in Fabric. BTW, why don't we use any package management tool?
kostas (Mon, 30 Oct 2017 09:12:04 GMT):
Huh. Interesting. Have you given this a go with the latest version of `viper` to see if it works as expected?
kostas (Mon, 30 Oct 2017 09:13:18 GMT):
> BTW, why don't we use any package management tool?
We use `govendor`. (But as your point above shows, we could probably do a better job managing our dependencies.)
guoger (Mon, 30 Oct 2017 15:04:20 GMT):
@kostas yep, but hitting some other issues:
- latest viper coverts all keys to lower case. To generate the genesis block, we should use `twoorgsorderergenesis` instead of `TwoOrgsOrdererGenesis`
- orderer crashes with `Orderer capability v1.1 is required but not supported`
But at least it's parses correctly
jyellick (Mon, 30 Oct 2017 15:16:43 GMT):
@guoger Sounds like you have this under control, I won't investigate then?
guoger (Mon, 30 Oct 2017 15:18:47 GMT):
@jyellick yeah, I think I find the cause, will file a jira later and submit a patch (probably need to figure out the pkg mgmt first)
kostas (Mon, 30 Oct 2017 15:34:15 GMT):
> Thanks, I was discussing this a bit on Friday with @kostas directly, wanted to let it percolate a bit this weekend, will try to write something up for you soon
Regarding this, and just so that we can get the discussion going:
kostas (Mon, 30 Oct 2017 15:39:38 GMT):
(Was about to suggest that we expand the `HeaderType` enum by adding a generic `Isolated` option, but then the peer would be unable to tell that they're dealing with a resource update message. So scratch that.)
kostas (Mon, 30 Oct 2017 15:44:17 GMT):
What are our thoughts regarding using the `extension` field of the `ChannelHeader` for this? I'm not the biggest fan because you're supposed to deserialize the value based on the header type. We could always, however, unmarshal against a proto message defined for isolation (to check if the message is to be isolated), besides the type-based checks.
kostas (Mon, 30 Oct 2017 15:44:17 GMT):
What are our thoughts on using the `extension` field of the `ChannelHeader` for this? I'm not the biggest fan because you're supposed to deserialize the value based on the header type. We could always, however, unmarshal against a proto message defined for isolation (to check if the message is to be isolated), besides the type-based checks.
kostas (Mon, 30 Oct 2017 15:44:42 GMT):
As I said, not the biggest fan. LMK what other options you're considering.
guoger (Mon, 30 Oct 2017 15:56:37 GMT):
https://jira.hyperledger.org/browse/FAB-6803
jyellick (Mon, 30 Oct 2017 15:57:33 GMT):
My first reaction was to add something very generic sounding like `map
jyellick (Mon, 30 Oct 2017 15:58:08 GMT):
@guoger https://gerrit.hyperledger.org/r/#/c/14813/
guoger (Mon, 30 Oct 2017 16:00:35 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=G2bWSos4SQzmuCtfs) @jyellick ah, I see, I marked mine as duplicate
guoger (Mon, 30 Oct 2017 16:02:02 GMT):
oh, they address similar issue but https://gerrit.hyperledger.org/r/#/c/14813 doesn't really solve the entire problem
jyellick (Mon, 30 Oct 2017 16:02:22 GMT):
I _think_ this is related to https://gerrit.hyperledger.org/r/#/c/12533/ which I was concerned would break us in unexpected ways
guoger (Mon, 30 Oct 2017 16:07:24 GMT):
I just reopened mine, feel free to leave any comment there if you think it's indeed duplicate. I'll revisit this my tomorrow, thx folks
kostas (Mon, 30 Oct 2017 19:38:27 GMT):
> My first reaction was to add something very generic sounding like `map
kostas (Mon, 30 Oct 2017 19:38:38 GMT):
(`processing_hints` is a horrible name though.)
guoger (Tue, 31 Oct 2017 04:36:03 GMT):
@jyellick I'm reading your comment in https://gerrit.hyperledger.org/r/#/c/13963/12/orderer/consensus/kafka/chain.go@54
I agree with you for both cases, just wanna make sure we share the same understanding:
- for the first case, it's due to the fact that we advance `LastOriginalOffsetProcessed` (LOOP) for *both regular and config messages*, however we only advance `LastResubmittedConfigOffset` (LRCO) for *config messages*. Therefore, if a regular msg is resubmitted and reprocessed after a resubmitted config msg, we have `LOOP > LRCO`. It's same to simple convert `LRCO==LOOP` to `LRCO<=LOOP` as if we have already processed a message with `LOOP = x`, it's certain that all resubmitted msg whose original offset is lower than `x` should've been reprocessed.
- for the second, the root cause is notorious non-determinism. The `LRCO` is advanced in orderer who resubmits the msg, but *not advanced* in those who don't. However, I think converting the equation to less-equal is probably not sufficient in this case, as we will be committing blocks with different metadata. I think a simply fix would be to advance `LRCO` if `OO > LRCO` when a resubmitted config msg is received.
cc @kostas @sanchezl
guoger (Tue, 31 Oct 2017 04:36:03 GMT):
@jyellick I'm reading your comment in https://gerrit.hyperledger.org/r/#/c/13963/12/orderer/consensus/kafka/chain.go@54
I agree with you for both cases, just wanna make sure we share the same understanding:
- for the first case, it's due to the fact that we advance `LastOriginalOffsetProcessed` (LOOP) for *both regular and config messages*, however we only advance `LastResubmittedConfigOffset` (LRCO) for *config messages only*. Therefore, if a regular msg is resubmitted and reprocessed *after* a resubmitted config msg, we have `LOOP > LRCO`. It's safe to simply convert `LRCO==LOOP` to `LRCO<=LOOP` as if we have already processed a message with `LOOP = x`, it's certain that all resubmitted msg whose original offset is lower than `x` should've been reprocessed.
- for the second, the root cause is notorious non-determinism. The `LRCO` is advanced in orderer who resubmits the msg, but *not advanced* in those who don't. However, I think converting the equation to less-equal is probably not sufficient in this case, as we will be committing blocks with different metadata. I think a simply fix would be to advance `LRCO` if `OO > LRCO` when a resubmitted config msg is received.
cc @kostas @sanchezl
guoger (Tue, 31 Oct 2017 04:36:03 GMT):
@jyellick I'm reading your comment in https://gerrit.hyperledger.org/r/#/c/13963/12/orderer/consensus/kafka/chain.go@54
I agree with you for both cases, just wanna make sure we share the same understanding:
- for the first case, it's due to the fact that we advance `LastOriginalOffsetProcessed` (LOOP) for *both regular and config messages*, however we only advance `LastResubmittedConfigOffset` (LRCO) for *config messages only*. Therefore, if a regular msg is resubmitted and reprocessed *after* a resubmitted config msg, we have `LOOP > LRCO`. It's safe to simply convert `LRCO==LOOP` to `LRCO<=LOOP` as if we have already processed a message with `LOOP = x`, it's certain that all resubmitted config msg whose original offset is lower than `x` should've been reprocessed.
- for the second, the root cause is notorious non-determinism. The `LRCO` is advanced in orderer who resubmits the msg, but *not advanced* in those who don't. However, I think converting the equation to less-equal is probably not sufficient in this case, as we will be committing blocks with different metadata. I think a simply fix would be to advance `LRCO` if `OO > LRCO` when a resubmitted config msg is received.
cc @kostas @sanchezl
guoger (Tue, 31 Oct 2017 04:36:03 GMT):
@jyellick I'm reading your comment in https://gerrit.hyperledger.org/r/#/c/13963/12/orderer/consensus/kafka/chain.go@54
I agree with you for both cases, just wanna make sure we share the same understanding:
- for the first case, it's due to the fact that we advance `LastOriginalOffsetProcessed` (LOOP) for *both regular and config messages*, however we only advance `LastResubmittedConfigOffset` (LRCO) for *config messages only*. Therefore, if a regular msg is resubmitted and reprocessed *after* a resubmitted config msg, we have `LOOP > LRCO`. It's safe to simply convert `LRCO==LOOP` to `LRCO<=LOOP` as if we have already processed a message with `LOOP = x`, it's certain that all resubmitted config msg whose original offset is lower than `x` should've been reprocessed.
- for the second, the root cause is notorious non-determinism. The `LRCO` is advanced in orderer who resubmits the msg, but *not advanced* in those who don't. However, I think converting the equation to less-equal is probably not sufficient in this case, as we will be committing blocks with different metadata (LRCO). I think a simply fix would be to advance `LRCO` if `OO > LRCO` when a resubmitted config msg is received.
cc @kostas @sanchezl
guoger (Tue, 31 Oct 2017 04:36:03 GMT):
@jyellick I'm reading your comment in https://gerrit.hyperledger.org/r/#/c/13963/12/orderer/consensus/kafka/chain.go@54
I agree with you for both cases, just wanna make sure we share the same understanding:
- for the first case, it's due to the fact that we advance `LastOriginalOffsetProcessed` (LOOP) for *both regular and config messages*, however we only advance `LastResubmittedConfigOffset` (LRCO) for *config messages only*. Therefore, if a regular msg is resubmitted and reprocessed *after* a resubmitted config msg, we have `LOOP > LRCO`. It's safe to simply convert `LRCO==LOOP` to `LRCO<=LOOP` as if we have already processed a message with `LOOP = x`, it's certain that all resubmitted config msg whose original offset is lower than `x` should've been reprocessed.
- for the second, the root cause is notorious non-determinism. The `LRCO` is advanced in orderer who resubmits the msg, but *not advanced* in those who don't. However, I think converting the equation to less-equal is probably not sufficient in this case, as we will be committing blocks with different metadata (LRCO). A simple fix would be to advance `LRCO` if `OO > LRCO` when a resubmitted config msg is received.
cc @kostas @sanchezl
jyellick (Tue, 31 Oct 2017 05:15:22 GMT):
> However, I think converting the equation to less-equal is probably not sufficient in this case, as we will be committing blocks with different metadata (LRCO).
@guoger Why is this a problem? Block metadata is not part of the hash, so it is okay if they are not consistent across the network (so long as the block data contents are)
guoger (Tue, 31 Oct 2017 05:34:19 GMT):
@jyellick when a new orderer joins the network, it replays all messages in the kafka to catch up with others. I thought it never advances LRCO in this case?
jyellick (Tue, 31 Oct 2017 05:46:21 GMT):
Why would it not?
jyellick (Tue, 31 Oct 2017 05:46:21 GMT):
~Why would it not?~ Sorry, I was thinking LOOP not LRCO
jyellick (Tue, 31 Oct 2017 05:46:21 GMT):
~Why would it not?~ Sorry, I was thinking LOOP not LRCO. What would be the harm in LRCO never advancing? So long as LOOP then you meet `LRCO<=LOOP`
jyellick (Tue, 31 Oct 2017 05:46:21 GMT):
~Why would it not?~ Sorry, I was thinking LOOP not LRCO. What would be the harm in LRCO never advancing? So long as LOOP advances, then you meet `LRCO<=LOOP`
guoger (Tue, 31 Oct 2017 06:13:06 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=38tk9vnCirevFkNnL) @jyellick True... LRCO is actually advanced during replay. New orderer would still resubmit a message if config seq is lagged, although it would be very stale for other orderers. But I feel enforcing consistent metadata across the network would be less error-prone.
guoger (Tue, 31 Oct 2017 06:13:51 GMT):
Also, this *replay* case is also an interesting area to pay attention.
jyellick (Tue, 31 Oct 2017 06:14:19 GMT):
Yes, certainly on replay, all wrong config seq messages are resubmitted
jyellick (Tue, 31 Oct 2017 06:14:56 GMT):
All OSNs will immediately ignore them (as will the submitting one when it processes them) because the last original offset processed will be so far off
jyellick (Tue, 31 Oct 2017 06:15:21 GMT):
But it will pollute the Kafka logs. Still, the overhead I would think should be negligable
jyellick (Tue, 31 Oct 2017 06:15:35 GMT):
Especially with the locking mechanism in place from this CR, hopefully the number of resubmitted messages is quite low.
guoger (Tue, 31 Oct 2017 06:24:49 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=RPyRqLQxBHwoF8fKB) @jyellick yes that's true.
Still, what's your thought on this?
> But I feel enforcing consistent metadata across the network would be less error-prone.
guoger (Tue, 31 Oct 2017 06:25:12 GMT):
Also, we don't block ingress msg while a brand new orderer is catching up, where resubmission could very likely happen...
kostas (Tue, 31 Oct 2017 09:14:09 GMT):
@guoger: In the first case, what is the sequence of messages that the OSN sees? Let's start with the config message that gets resubmitted with an offset of 5 and take it from there
kostas (Tue, 31 Oct 2017 09:14:09 GMT):
@guoger: In the first case, what is the sequence of messages that the OSN sees? Let's start with the config message that gets resubmitted with an offset of 5 and take it from there.
guoger (Tue, 31 Oct 2017 09:23:28 GMT):
msg_seq.jpeg
guoger (Tue, 31 Oct 2017 09:23:48 GMT):
@kostas this is the graph I came up with to show the message seq
kostas (Tue, 31 Oct 2017 09:23:54 GMT):
*looking
guoger (Tue, 31 Oct 2017 09:28:43 GMT):
I'm always curious.. what's your timezone? :P
kostas (Tue, 31 Oct 2017 09:29:57 GMT):
EDT.
kostas (Tue, 31 Oct 2017 09:31:20 GMT):
(It is admittedly impressive if you consider that I'm in bed by 9pm.)
kostas (Tue, 31 Oct 2017 09:31:20 GMT):
(It is admittedly not impressive if you consider that I'm in bed by 9pm.)
guoger (Tue, 31 Oct 2017 09:32:05 GMT):
I had an impression that you are literally working around the clock
guoger (Tue, 31 Oct 2017 09:33:00 GMT):
sometimes I couldn't really tell if you stay up late or wake up early...
kostas (Tue, 31 Oct 2017 09:40:01 GMT):
Shouldn't the inequality by `<=`?
kostas (Tue, 31 Oct 2017 09:40:16 GMT):
Looking at the comment where we suggest it's turned to `>=`?
kostas (Tue, 31 Oct 2017 09:40:16 GMT):
Looking at the comment where we suggest it's turned to `>=`.
kostas (Tue, 31 Oct 2017 09:40:16 GMT):
Looking at the comment where the suggestion is to switch to `>=`.
guoger (Tue, 31 Oct 2017 09:41:02 GMT):
I believe it's a typo. It's `<=` in the conversation here
kostas (Tue, 31 Oct 2017 09:41:17 GMT):
Ah, let me look again.
kostas (Tue, 31 Oct 2017 09:41:22 GMT):
Good drawing BTW, thanks.
guoger (Tue, 31 Oct 2017 09:47:50 GMT):
OO is Original Offset BTW
kostas (Tue, 31 Oct 2017 09:53:57 GMT):
> Also, we don't block ingress msg while a brand new orderer is catching up, where resubmission could very likely happen...
Why do you say this?
kostas (Tue, 31 Oct 2017 09:53:57 GMT):
> Also, we don't block ingress msg while a brand new orderer is catching up, where resubmission could very likely happen...
Hm, what makes you say this?
kostas (Tue, 31 Oct 2017 09:54:23 GMT):
You are still going to hit line 787, right?
jyellick (Tue, 31 Oct 2017 14:20:55 GMT):
@kostas
> Hm, what makes you say this?
The orderer does not/cannot know that the config sequence will be advancing in the future. Consider a simple chain, genesis, 100k txs, then reconfiguration to sequence 2. While the new orderer is catching up, he will have config sequence 1 until he has replayed all 100k txes, then he will move to seq 2. If at any point during this process he receives a `Broadcast`, he will forward it with config seq 0. Which, will obviously need to be replayed.
jyellick (Tue, 31 Oct 2017 14:22:13 GMT):
I'm not sure that this is so much of an issue though. I believe our recommended procedure for adding an orderer would be to:
1. Bootstrap the orderer with the genesis block, or some backup of the ledger.
2. Wait for the new orderer to come completely in sync with the rest of the OSNs
3. Modify the configuration of each channel to add the new OSN to the list of OSNs.
jyellick (Tue, 31 Oct 2017 14:22:26 GMT):
So, hopefully, no one is actually referencing your new orderer until he is all the way caught up
kostas (Tue, 31 Oct 2017 15:04:03 GMT):
ACK, agreed. Not really an issue.
kostas (Tue, 31 Oct 2017 15:04:03 GMT):
ACK, understood. I agree, not really an issue.
simsc (Tue, 31 Oct 2017 21:28:56 GMT):
Has joined the channel.
guoger (Wed, 01 Nov 2017 01:39:41 GMT):
yep, I just wanted to point it out, and maybe we should document it somewhere. Also, I feel it should be taken into consideration while designing the ledger-prune feature. cc @sanchezl
guoger (Wed, 01 Nov 2017 13:29:05 GMT):
@sanchezl if you need some help reviewing/collaborating on the doc, let me know :)
guoger (Fri, 03 Nov 2017 10:22:06 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=LiCMzsB9BnGzg6TyX) @kostas @jyellick I still want some suggestions on this, so that I could update https://gerrit.hyperledger.org/r/c/13963, thx a lot!!!
kostas (Fri, 03 Nov 2017 10:31:41 GMT):
Please remind me -- what does consistent metadata in this case buys you?
kostas (Fri, 03 Nov 2017 10:31:41 GMT):
Please remind me -- what does consistent metadata in this case buy you?
guoger (Fri, 03 Nov 2017 10:47:56 GMT):
ah, let me elaborate a bit. It is to address the second case @jyellick pointed out in that CR:
> Consider two OSNs in a consistent state, both of which receive a config message which is out of date (old config sequence). OSN1 processes the config and finds it invalid for some non-deterministic reason (say cert expiration), while OSN2 processes the config and resubmits it. Then on commit, lastOriginalOffsetProcessed advances on both, but the lastResubmittedConfigOffset is still old on OSN1.
The root cause is notorious non-determinism. The `LRCO` is advanced in orderer who resubmits the msg, but *not advanced* in those who don't. However, I think simply converting the equation to less-equal is probably not sufficient in this case, as *we will be committing blocks with different metadata (LRCO)*. A simple fix would be to advance `LRCO` if `OO > LRCO` when a resubmitted config msg is received.
guoger (Fri, 03 Nov 2017 10:47:56 GMT):
ah, let me elaborate a bit. It is to address the second case @jyellick pointed out in that CR:
> Consider two OSNs in a consistent state, both of which receive a config message which is out of date (old config sequence). OSN1 processes the config and finds it invalid for some non-deterministic reason (say cert expiration), while OSN2 processes the config and resubmits it. Then on commit, lastOriginalOffsetProcessed advances on both, but the lastResubmittedConfigOffset is still old on OSN1.
The root cause is notorious non-determinism. The `LRCO` is advanced in orderer who resubmits the msg, but *not advanced* in those who don't. However, I think simply converting the equation to less-equal is probably not sufficient in this case, as *we will be committing blocks with different metadata (LRCO)*.
My proposal is to advance `LRCO` if `OO > LRCO` when a resubmitted config msg is received.
guoger (Fri, 03 Nov 2017 10:48:00 GMT):
@kostas
guoger (Fri, 03 Nov 2017 10:50:35 GMT):
This is really just a nit, but I want to point out that we probably should alway enforce consistent metadata across network, even if it doesn't harm not to do so. Essentially I feel our code is getting more and more error-prone as we keep adding more fields and logics to check them.
guoger (Fri, 03 Nov 2017 11:58:15 GMT):
Also I replied in https://jira.hyperledger.org/browse/FAB-6803?focusedCommentId=34306&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-34306
kostas (Sat, 04 Nov 2017 13:23:23 GMT):
> My proposal is to advance `LRCO` if `OO > LRCO` when a resubmitted config msg is received.
@guoger: If it's trivial to produce this patchset, do so and we can review its merits appropriately.
kostas (Sat, 04 Nov 2017 13:23:27 GMT):
Inconsistent metadata across the network, esp. on these esoteric fields, is very low on my list of concerns though.
kostas (Sat, 04 Nov 2017 13:23:44 GMT):
This will hopefully be the last time we mess around with adding new metadata fields and checks for some time, so if we what we have so far works, we're good.
kostas (Wed, 08 Nov 2017 02:56:50 GMT):
@guoger: Thinking aloud and I'm tired, but do you think there's a meaningful (i.e. non-boring, easy-to-follow) playback that can be made out of your Kafka non-determinism work? Give it some thought, and if you believe this can be done, let's consider adding it here: https://wiki.hyperledger.org/projects/fabric/playbacks
guoger (Wed, 08 Nov 2017 03:59:47 GMT):
@kostas interesting... I didn't know there's a playback :P I think it could be done, but probably not an one-hour talk. I think 30 min would be sufficient for this..
kostas (Wed, 08 Nov 2017 13:14:36 GMT):
30 minutes is great.
sanchezl (Thu, 09 Nov 2017 04:58:13 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=3KsbKN4cZLHN2eHWi) @kostas @gouger, I've been performing some tests on v1.0.4 where I move chains from one orderer to another. I've discovered that if I move a chain, without the index that contains the offsets of the blocks in the chain storage, the indexes do not match, meaning that there is some variation in the blocks across the two orderers. I would suspect maybe a difference in the metadata (in v1.0.4 we only add last_offset_persisted). Could the last_offset_persisted metadata (a uint64 persisted as a varint) be different on the two orderers?
sanchezl (Thu, 09 Nov 2017 04:58:13 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=3KsbKN4cZLHN2eHWi) @kostas @gouger, I've been performing some tests on v1.0.4 where I move chains from one orderer to another. I've discovered that if I move a chain, without the index that contains the offsets of the blocks in the chain storage, the indexes do not match, meaning that there is some variation in the blocks across the two orderers. I would suspect maybe a difference in the metadata (in v1.0.4 we only add last_offset_persisted). Could the last_offset_persisted metadata (a uint64 persisted as a varint) be different on the two orderers? If so,we might already have inconsistent metadata. I don't see that as a problem though.
guoger (Thu, 09 Nov 2017 08:11:32 GMT):
@sanchezl AFAIK, `last_offset_persisted` should be consistent across the OSNs. How do you move chains? copying ledger files elsewhere and bootstrap an orderer there?
sanchezl (Thu, 09 Nov 2017 10:55:15 GMT):
@guoger , yes copying files. I’m looking into it with the ledger folks, might be something else going on with the ledger impl itself.
kostas (Thu, 09 Nov 2017 12:46:13 GMT):
@sanchezl: Interesting. I was under the impression that you can simply delete the index folder and it will be rebuilt automatically: https://chat.hyperledger.org/channel/fabric-ledger?msg=eA5khDwmhjBbFkHY2
kostas (Thu, 09 Nov 2017 12:47:07 GMT):
Then you'd end up with an index that's consistent with your orderer's ledger. I am guessing that is not what you're seeing? (Or are you trying a slightly different variation?_
kostas (Thu, 09 Nov 2017 12:47:07 GMT):
Then you'd end up with an index that's consistent with your orderer's ledger. I am guessing that is not what you're seeing? (Or are you trying a slightly different variation?)
sanchezl (Thu, 09 Nov 2017 14:56:41 GMT):
@kostas: I was trying a variation where I copied the chain but did not delete the existing index (so no index rebuild). While I would not recommend those exact steps, the ledger team felt that the index should of still been valid for the blocks thats existed before the copy, but it wasn't. Deleting the index first, would solve the problem, as the index, as of version v1.0.4, would rebuild automatically.
kostas (Thu, 09 Nov 2017 14:57:58 GMT):
Got it, thank you.
jetsginza (Mon, 13 Nov 2017 02:33:21 GMT):
Has joined the channel.
guoger (Tue, 14 Nov 2017 01:36:30 GMT):
@jyellick I saw your comments in [blocking ingress message patch](https://gerrit.hyperledger.org/r/#/c/13963/), thx! I'll wait for other folks to give another round of review and rebase/revise altogether, cc @kostas @sanchezl
qiang0723 (Thu, 16 Nov 2017 06:57:20 GMT):
Has joined the channel.
kostas (Tue, 21 Nov 2017 22:14:42 GMT):
My mind is slightly blown by the fact that (a) such a library exists, (b) @sanchezl found it, and (c) it fits our needs for the Kafka versioning problem perfectly: https://github.com/hashicorp/go-version and https://gerrit.hyperledger.org/r/c/15643/
jackeyliliang (Fri, 24 Nov 2017 02:59:52 GMT):
Has joined the channel.
arjanvaneersel (Mon, 27 Nov 2017 17:08:40 GMT):
Has joined the channel.
kostas (Wed, 29 Nov 2017 16:52:27 GMT):
```$ protoc --go_out=. *.proto
app.pb.go: Tried to write the same file twice.```
kostas (Wed, 29 Nov 2017 16:52:37 GMT):
Anyone bumped into this before? My Google-fu fails me.
kostas (Wed, 29 Nov 2017 17:26:06 GMT):
What am I missing here?
kostas (Wed, 29 Nov 2017 17:26:25 GMT):
```~/Go/src/github.com/kchristidis/float (master)* $ protoc --version
libprotoc 3.5.0
~/Go/src/github.com/kchristidis/float (master)* $ protoc app.proto
~/Go/src/github.com/kchristidis/float (master)* $ ls -lT app.pb.go
-rw-r--r-- 1 kchrist staff 6529 Nov 29 12:24:27 2017 app.pb.go
~/Go/src/github.com/kchristidis/float (master)* $
~/Go/src/github.com/kchristidis/float (master)* $
~/Go/src/github.com/kchristidis/float (master)* $ rm app.pb.go
~/Go/src/github.com/kchristidis/float (master)* $ protoc --go_out=. app.proto
app.pb.go: Tried to write the same file twice.
~/Go/src/github.com/kchristidis/float (master)* $ ls -lT app.pb.go
ls: app.pb.go: No such file or directory
~/Go/src/github.com/kchristidis/float (master)* $
~/Go/src/github.com/kchristidis/float (master)* $
~/Go/src/github.com/kchristidis/float (master)* $ protoc --go_out=plugins=grpc:. app.proto
app.pb.go: Tried to write the same file twice.
~/Go/src/github.com/kchristidis/float (master)* $ ls -lT app.pb.go
ls: app.pb.go: No such file or directory```
kostas (Wed, 29 Nov 2017 17:26:25 GMT):
```~/Go/src/github.com/kchristidis/float (master)* $ go version
go version go1.9.2 darwin/amd64
~/Go/src/github.com/kchristidis/float (master)* $ protoc --version
libprotoc 3.5.0
~/Go/src/github.com/kchristidis/float (master)* $ protoc app.proto
~/Go/src/github.com/kchristidis/float (master)* $ ls -lT app.pb.go
-rw-r--r-- 1 kchrist staff 6529 Nov 29 12:24:27 2017 app.pb.go
~/Go/src/github.com/kchristidis/float (master)* $
~/Go/src/github.com/kchristidis/float (master)* $
~/Go/src/github.com/kchristidis/float (master)* $ rm app.pb.go
~/Go/src/github.com/kchristidis/float (master)* $ protoc --go_out=. app.proto
app.pb.go: Tried to write the same file twice.
~/Go/src/github.com/kchristidis/float (master)* $ ls -lT app.pb.go
ls: app.pb.go: No such file or directory
~/Go/src/github.com/kchristidis/float (master)* $
~/Go/src/github.com/kchristidis/float (master)* $
~/Go/src/github.com/kchristidis/float (master)* $ protoc --go_out=plugins=grpc:. app.proto
app.pb.go: Tried to write the same file twice.
~/Go/src/github.com/kchristidis/float (master)* $ ls -lT app.pb.go
ls: app.pb.go: No such file or directory```
kostas (Wed, 29 Nov 2017 17:27:15 GMT):
Notice that the first (and only) successful invocation of `protoc` gives me an `app.pb.go` file that, suprisingly, includes the stub for the gRPC service definition in `app.proto`.
kostas (Wed, 29 Nov 2017 17:28:34 GMT):
It gets even weirder:
kostas (Wed, 29 Nov 2017 17:28:37 GMT):
```~/Go/src/github.com/kchristidis/float (master)* $ protoc --go_out=test app.proto
~/Go/src/github.com/kchristidis/float (master)* $ ls -lT test/app.pb.go
-rw-r--r-- 1 kchrist staff 3527 Nov 29 12:28:11 2017 test/app.pb.go```
kostas (Wed, 29 Nov 2017 17:29:32 GMT):
This invocation with `--go_out` works as expected. And notice that `app.pb.go` is smaller, _as expected_, since it doesn't contain the gRPC service stub.
kostas (Wed, 29 Nov 2017 17:29:59 GMT):
Also as expected:
kostas (Wed, 29 Nov 2017 17:30:04 GMT):
```~/Go/src/github.com/kchristidis/float (master)* $ protoc --go_out=plugins=grpc:test app.proto
~/Go/src/github.com/kchristidis/float (master)* $ ls -lT test/app.pb.go
-rw-r--r-- 1 kchrist staff 6529 Nov 29 12:29:49 2017 test/app.pb.go
```
jyellick (Wed, 29 Nov 2017 17:30:10 GMT):
Agreed that it's all very bizarre. Have you tried via `make protos`? Curious if it's a local binary issue for you
jyellick (Wed, 29 Nov 2017 17:30:35 GMT):
(Or even simply executing from inside the fabric image which does so using that version of proto)
kostas (Wed, 29 Nov 2017 17:31:41 GMT):
Am I mistaken, or `make protos` will only work only for the `*.proto` definition inside the `fabric` dir?
kostas (Wed, 29 Nov 2017 17:31:41 GMT):
Am I mistaken, or `make protos` will only work only for the `*.proto` definitions inside the `fabric` dir?
kostas (Wed, 29 Nov 2017 17:33:23 GMT):
Meanwhile, I have uninstalled and reinstalled `protoc` and `protoc-gen-go` on my machine w/ no luck.
jyellick (Wed, 29 Nov 2017 17:33:45 GMT):
> Am I mistaken, or `make protos` will only work only for the `*.proto` definitions inside the `fabric` dir?
Correct
kostas (Wed, 29 Nov 2017 17:33:47 GMT):
No point in wasting anybody else's cycles here, and I guess I'll figure it out eventually, was just wondering whether I'm missing something obvious.
jyellick (Wed, 29 Nov 2017 17:34:35 GMT):
Have you attempted in Vagrant or on some other host? I could try running it on my local machine if want to send me the proto
kostas (Wed, 29 Nov 2017 17:38:57 GMT):
Ah, let me give it a shot in Vagrant. It does piss me off though that I can't figure it out yet. (And even if I could make it work with `make protos` or Vagrant, the downside is that this would prevent me from iterating as fast as I'd like.) Working out that raft implementation but I'm treating as an entirely different project for now to cut down on unneeded dependencies.
kostas (Wed, 29 Nov 2017 17:38:57 GMT):
Ah, let me give it a shot in Vagrant. It does piss me off though that I can't figure it out yet. (And even if I could make it work with `make protos` or Vagrant, the downside is that this would prevent me from iterating as fast as I'd like.) Working on that raft implementation but I'm treating as an entirely different project for now to cut down on unneeded dependencies.
kostas (Wed, 29 Nov 2017 17:38:57 GMT):
Ah, let me give it a shot in Vagrant. It does piss me off though that I can't figure it out yet. (And even if I could make it work with `make protos` or Vagrant, the downside is that this would prevent me from iterating as fast as I'd like.) Working on that raft implementation but I'm treating it as an entirely different project for now to cut down on unneeded dependencies.
kostas (Wed, 29 Nov 2017 17:47:55 GMT):
Argh, works w/o issues in Vagrant:
kostas (Wed, 29 Nov 2017 17:48:04 GMT):
```ubuntu@hyperledger-devenv:5b38cbc:/opt/gopath/src/github.com/hyperledger/fabric$ go version
go version go1.9 linux/amd64
ubuntu@hyperledger-devenv:5b38cbc:/opt/gopath/src/github.com/hyperledger/fabric$ protoc --version
libprotoc 3.5.0
ubuntu@hyperledger-devenv:5b38cbc:/opt/gopath/src/github.com/hyperledger/fabric$ protoc --go_out=. app.proto
ubuntu@hyperledger-devenv:5b38cbc:/opt/gopath/src/github.com/hyperledger/fabric$ ls -la app.pb.go
-rw-r--r-- 1 ubuntu ubuntu 4025 Nov 29 17:46 app.pb.go
ubuntu@hyperledger-devenv:5b38cbc:/opt/gopath/src/github.com/hyperledger/fabric$ protoc --go_out=plugins=grpc:. app.proto
ubuntu@hyperledger-devenv:5b38cbc:/opt/gopath/src/github.com/hyperledger/fabric$ ls -la app.pb.go
-rw-r--r-- 1 ubuntu ubuntu 7027 Nov 29 17:47 app.pb.go
ubuntu@hyperledger-devenv:5b38cbc:/opt/gopath/src/github.com/hyperledger/fabric$```
kostas (Wed, 29 Nov 2017 17:48:04 GMT):
```ubuntu@hyperledger-devenv:5b38cbc:/opt/gopath/src/github.com/hyperledger/fabric$ go version
go version go1.9 linux/amd64
ubuntu@hyperledger-devenv:5b38cbc:/opt/gopath/src/github.com/hyperledger/fabric$ protoc --version
libprotoc 3.5.0
ubuntu@hyperledger-devenv:5b38cbc:/opt/gopath/src/github.com/hyperledger/fabric$ protoc --go_out=. app.proto
ubuntu@hyperledger-devenv:5b38cbc:/opt/gopath/src/github.com/hyperledger/fabric$ ls -la app.pb.go
-rw-r--r-- 1 ubuntu ubuntu 4025 Nov 29 17:46 app.pb.go
ubuntu@hyperledger-devenv:5b38cbc:/opt/gopath/src/github.com/hyperledger/fabric$ protoc --go_out=plugins=grpc:. app.proto
ubuntu@hyperledger-devenv:5b38cbc:/opt/gopath/src/github.com/hyperledger/fabric$ ls -la app.pb.go
-rw-r--r-- 1 ubuntu ubuntu 7027 Nov 29 17:47 app.pb.go
```
kostas (Wed, 29 Nov 2017 17:48:56 GMT):
Two differences compared to my local environment: host OS and Go version (I'm on 1.9.2)
kostas (Wed, 29 Nov 2017 17:49:17 GMT):
Anyway, thanks for the suggestion @jyellick. At least I can hack my way around it like this for the time being.
kostas (Wed, 29 Nov 2017 17:49:44 GMT):
@guoger and @sanchezl I know both of you use Macs -- I'm curious if you're having issues with `protoc` as well.
sanchezl (Wed, 29 Nov 2017 19:53:01 GMT):
@kostas,
sanchezl (Wed, 29 Nov 2017 19:53:01 GMT):
@kostas, I don't have a protoc compiled in my env, but here is a similar flow using the `fabric-buildenv` docker images:
```Firehawk:kostas sanchezl$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.13.1
BuildVersion: 17B48
Firehawk:kostas sanchezl$ go version
go version go1.9.2 darwin/amd64
Firehawk:kostas sanchezl$ protoc --version
-bash: protoc: command not found
Firehawk:kostas sanchezl$ docker run -i --rm -v $PWD:/opt/gopath/src/github.com/hyperledger/fabric -w /opt/gopath/src/github.com/hyperledger/fabric hyperledger/fabric-buildenv protoc --version
libprotoc 3.1.0
Firehawk:kostas sanchezl$ docker run -i --rm -v $PWD:/opt/gopath/src/github.com/hyperledger/fabric -w /opt/gopath/src/github.com/hyperledger/fabric hyperledger/fabric-buildenv protoc app.version
Missing output directives.
Firehawk:kostas sanchezl$ ls -l
total 8
-rw-r--r--@ 1 sanchezl staff 298 Nov 29 14:05 app.proto
Firehawk:kostas sanchezl$ docker run -i --rm -v $PWD:/opt/gopath/src/github.com/hyperledger/fabric -w /opt/gopath/src/github.com/hyperledger/fabric hyperledger/fabric-buildenv protoc app.proto
Missing output directives.
Firehawk:kostas sanchezl$ docker run -i --rm -v $PWD:/opt/gopath/src/github.com/hyperledger/fabric -w /opt/gopath/src/github.com/hyperledger/fabric hyperledger/fabric-buildenv protoc --go_out=. app.proto
Firehawk:kostas sanchezl$ ls -l
total 16
-rw-r--r-- 1 sanchezl staff 4031 Nov 29 14:50 app.pb.go
-rw-r--r--@ 1 sanchezl staff 298 Nov 29 14:05 app.proto
Firehawk:kostas sanchezl$ rm app.pb.go
Firehawk:kostas sanchezl$ ls -l
total 8
-rw-r--r--@ 1 sanchezl staff 298 Nov 29 14:05 app.proto
Firehawk:kostas sanchezl$ docker run -i --rm -v $PWD:/opt/gopath/src/github.com/hyperledger/fabric -w /opt/gopath/src/github.com/hyperledger/fabric hyperledger/fabric-buildenv protoc --go_out=. app.proto
Firehawk:kostas sanchezl$ ls -l
total 16
-rw-r--r-- 1 sanchezl staff 4031 Nov 29 14:51 app.pb.go
-rw-r--r--@ 1 sanchezl staff 298 Nov 29 14:05 app.proto
Firehawk:kostas sanchezl$ docker run -i --rm -v $PWD:/opt/gopath/src/github.com/hyperledger/fabric -w /opt/gopath/src/github.com/hyperledger/fabric hyperledger/fabric-buildenv protoc --go_out=. app.proto
Firehawk:kostas sanchezl$ ls -l
total 16
-rw-r--r-- 1 sanchezl staff 4031 Nov 29 14:51 app.pb.go
-rw-r--r--@ 1 sanchezl staff 298 Nov 29 14:05 app.proto
Firehawk:kostas sanchezl$ docker run -i --rm -v $PWD:/opt/gopath/src/github.com/hyperledger/fabric -w /opt/gopath/src/github.com/hyperledger/fabric hyperledger/fabric-buildenv protoc --go_out=plugins=grpc:. app.proto
Firehawk:kostas sanchezl$ ls -l
total 24
-rw-r--r-- 1 sanchezl staff 7038 Nov 29 14:52 app.pb.go
-rw-r--r--@ 1 sanchezl staff 298 Nov 29 14:05 app.proto
Firehawk:kostas sanchezl$ ```
sanchezl (Wed, 29 Nov 2017 19:53:01 GMT):
@kostas, I don't have a protoc compiled in my env, but here is a similar flow using the `fabric-buildenv` docker images:
```Firehawk:kostas sanchezl$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.13.1
BuildVersion: 17B48
Firehawk:kostas sanchezl$ go version
go version go1.9.2 darwin/amd64
Firehawk:kostas sanchezl$ protoc --version
-bash: protoc: command not found
Firehawk:kostas sanchezl$ docker run -i --rm -v $PWD:/opt/gopath/src/github.com/hyperledger/fabric -w /opt/gopath/src/github.com/hyperledger/fabric hyperledger/fabric-buildenv protoc --version
libprotoc 3.1.0
Firehawk:kostas sanchezl$ ls -l
total 8
-rw-r--r--@ 1 sanchezl staff 298 Nov 29 14:05 app.proto
Firehawk:kostas sanchezl$ docker run -i --rm -v $PWD:/opt/gopath/src/github.com/hyperledger/fabric -w /opt/gopath/src/github.com/hyperledger/fabric hyperledger/fabric-buildenv protoc app.proto
Missing output directives.
Firehawk:kostas sanchezl$ docker run -i --rm -v $PWD:/opt/gopath/src/github.com/hyperledger/fabric -w /opt/gopath/src/github.com/hyperledger/fabric hyperledger/fabric-buildenv protoc --go_out=. app.proto
Firehawk:kostas sanchezl$ ls -l
total 16
-rw-r--r-- 1 sanchezl staff 4031 Nov 29 14:50 app.pb.go
-rw-r--r--@ 1 sanchezl staff 298 Nov 29 14:05 app.proto
Firehawk:kostas sanchezl$ rm app.pb.go
Firehawk:kostas sanchezl$ ls -l
total 8
-rw-r--r--@ 1 sanchezl staff 298 Nov 29 14:05 app.proto
Firehawk:kostas sanchezl$ docker run -i --rm -v $PWD:/opt/gopath/src/github.com/hyperledger/fabric -w /opt/gopath/src/github.com/hyperledger/fabric hyperledger/fabric-buildenv protoc --go_out=. app.proto
Firehawk:kostas sanchezl$ ls -l
total 16
-rw-r--r-- 1 sanchezl staff 4031 Nov 29 14:51 app.pb.go
-rw-r--r--@ 1 sanchezl staff 298 Nov 29 14:05 app.proto
Firehawk:kostas sanchezl$ docker run -i --rm -v $PWD:/opt/gopath/src/github.com/hyperledger/fabric -w /opt/gopath/src/github.com/hyperledger/fabric hyperledger/fabric-buildenv protoc --go_out=. app.proto
Firehawk:kostas sanchezl$ ls -l
total 16
-rw-r--r-- 1 sanchezl staff 4031 Nov 29 14:51 app.pb.go
-rw-r--r--@ 1 sanchezl staff 298 Nov 29 14:05 app.proto
Firehawk:kostas sanchezl$ docker run -i --rm -v $PWD:/opt/gopath/src/github.com/hyperledger/fabric -w /opt/gopath/src/github.com/hyperledger/fabric hyperledger/fabric-buildenv protoc --go_out=plugins=grpc:. app.proto
Firehawk:kostas sanchezl$ ls -l
total 24
-rw-r--r-- 1 sanchezl staff 7038 Nov 29 14:52 app.pb.go
-rw-r--r--@ 1 sanchezl staff 298 Nov 29 14:05 app.proto
Firehawk:kostas sanchezl$ ```
kostas (Wed, 29 Nov 2017 19:56:33 GMT):
Thanks Luis. I'll keep that flow in mind as well.
kostas (Wed, 29 Nov 2017 19:56:45 GMT):
I've also filed a bug against the golang/protobuf repo: https://github.com/golang/protobuf/issues/460
Russell-Columbia (Wed, 29 Nov 2017 23:01:10 GMT):
Has joined the channel.
jyellick (Thu, 30 Nov 2017 14:46:09 GMT):
One remaining items for the lifecycle work which has not been tackled yet, is to isolate the resource update txes into their own blocks. My initial thought had been to add something to the ChannelHeader protobuf, but it's not obvious what that would look like. The simplest mechanism would be to simply switch off of the transaction type in the header. I dislike this because it adds some, albeit very limited, knowledge of the peer processing model into the orderer. On the other hand, since it would not modify any protos and be internal only to the orderer code only, it's definitely simplest. Opinions?
jyellick (Thu, 30 Nov 2017 14:46:09 GMT):
One remaining items for the lifecycle work which has not been tackled yet, is to isolate the resource update txes into their own blocks. My initial thought had been to add something to the ChannelHeader protobuf, but it's not obvious what that would look like. The simplest mechanism would be to simply switch off of the transaction type in the header. I dislike this because it adds some, albeit very limited, knowledge of the peer processing model into the orderer. On the other hand, since it would not modify any protos and be internal only to the orderer code, it's definitely simplest. Opinions?
kostas (Thu, 30 Nov 2017 14:58:46 GMT):
@jyellick: So `if ChannelHeader.Type == PEER_RESOURCE_UPDATE`, proceed with isolation?
kostas (Thu, 30 Nov 2017 14:58:46 GMT):
@jyellick: So if `ChannelHeader.Type == PEER_RESOURCE_UPDATE`, proceed with isolation?
jyellick (Thu, 30 Nov 2017 14:59:14 GMT):
That is the "it's simple, but injects peer processing model" solution
kostas (Thu, 30 Nov 2017 15:03:58 GMT):
How about using the `extension` field in the `ChannelHeader` for this? Meaning, you define a protobuf message for isolated messages, and you check whether a non-nil extension deserializes to this `Isolated` message successfully. If it does, you cut it into its own batch. There may be other drawbacks to this approach (it'll definitely be slower for one), but I like it because doesn't leak any of the peer processing model logic into the orderer.
jyellick (Thu, 30 Nov 2017 15:17:41 GMT):
Unfortunately, I think the `extension` field is already used by the peer for additional chaincode header information. Since this header is used as the proposal header and then the channel header, I don't think we can modify that field.
jyellick (Thu, 30 Nov 2017 15:18:44 GMT):
Additionally, you can't count on protobuf deserialization to fail if it is not the correct type. If someone stored other information in there, there is no reason to expect it would not deserialize to the `Isolated` message type.
kostas (Thu, 30 Nov 2017 15:28:48 GMT):
> If someone stored other information in there, there is no reason to expect it would not deserialize to the `Isolated` message type.
Not sure I follow?
kostas (Thu, 30 Nov 2017 15:30:59 GMT):
Do we mean that given:
```
message Isolated {
bool val = 1;
}
message Custom {
bool val = 1;
}
```
A serialized `Isolated` message may unmarshal to `Custom` w/o issues?
kostas (Thu, 30 Nov 2017 15:30:59 GMT):
Do we mean that given:
```
message Isolated {
bool val = 1;
}
message Custom {
bool val = 1;
}
```
A serialized `Isolated` message may unmarshal to `Custom` w/o issues?
jyellick (Thu, 30 Nov 2017 15:34:29 GMT):
Absolutely
jyellick (Thu, 30 Nov 2017 15:34:38 GMT):
In fact, per your example it would _always_ deserialize without issues
kostas (Thu, 30 Nov 2017 15:34:57 GMT):
Yeah, this had not occured to me but makes perfect sense in hindsight.
jyellick (Thu, 30 Nov 2017 15:35:50 GMT):
In general, the only time you would see deserialization errors, are when the messages have the same field numbers, but different types. And the type encoding cannot be mistaken for another.
jyellick (Thu, 30 Nov 2017 15:35:50 GMT):
In general, the only time you would see deserialization errors, are when the messages have the same field numbers, but different field types. And the type encoding cannot be mistaken for another.
jyellick (Thu, 30 Nov 2017 15:38:01 GMT):
(For instance, I think you might be able to decode an int32 as a []byte, but perhaps not a string as an int32, but I'm not positive of the specifics)
kostas (Thu, 30 Nov 2017 15:38:42 GMT):
When do you need to settle on this? I dislike the idea of the orderer knowing that ` PEER_RESOURCE_UPDATE` is a thing and treating it as a special case, but if I can't come up with a better alternative, I'll shut up. (I do think it is a mistake though.)
kostas (Thu, 30 Nov 2017 15:38:42 GMT):
When do you need to settle on this? I dislike the idea of the orderer knowing that `PEER_RESOURCE_UPDATE` is a thing and treating it as a special case, but if I can't come up with a better alternative, I'll shut up. (I do think it is a mistake though.)
jyellick (Thu, 30 Nov 2017 15:39:19 GMT):
The official hard deadline is effectively 2 weeks to code completion.
jyellick (Thu, 30 Nov 2017 15:40:10 GMT):
I'd prefer to settle on this aspect sooner than later. I've been putting it off because I can't find a solution I like, but also because I expect that we, the orderer devs, can come to consensus on this issue much faster than the other pieces I have been pushing.
kostas (Thu, 30 Nov 2017 15:41:13 GMT):
OK, I will see if I can come up with something better by EOD tomorrow. If there are no better suggestions from anyone else as well, we roll with the HeaderType one.
guoger (Fri, 01 Dec 2017 07:29:20 GMT):
do we actually put timestamp on a block?
kostas (Fri, 01 Dec 2017 13:46:51 GMT):
@guoger: Clients may set the ChannelHeader's timestamp field when pushing an envelope the ordering service, but that's about it when it comes to timestamps included on blocks. There is block timestamp. Also, this JIRA issue might be of interest to you: https://jira.hyperledger.org/browse/FAB-1430
kostas (Fri, 01 Dec 2017 15:13:39 GMT):
Can we think of anything that may have caused a decrease of the ordering service's performance (as measured by tps) in this list of commits?
kostas (Fri, 01 Dec 2017 15:13:46 GMT):
```$ git log 863d7848d4c536ec4d5ff416f3522570e122ab66.. orderer
943f0ce574d3bcac7f9442e745aa7168e5992e9d [FAB-7105] Move orderer/common/util to common/util
784deb5feef8cbb0f76ab8434bc2452befa567d3 Merge "[FAB-7034] Configure orderer keepalive params"
d972b5453c99fd48f31e4120dc29d8ae4295e229 [FAB-7034] Configure orderer keepalive params
0c028df008b73aa606abbe0ee06b37a4f9854210 [FAB-5969] Block ingress msg for reprocessed msg
f7093140d47af369c9450cfb2c6171831d7bce8a [FAB-7044] Refactor gRPC server config code
576114618c57f662d10ee5cecb3e07d27ac9b89a [FAB-6840] Consolidate configtxapi to configtx
5d410fe106fc29403a7bbb0d1b15f8c279f346f6 [FAB-6839] configtx to directly utilize cb.Config
dcf36eb5bb454934fbaec9f80b2c22e177e54740 [FAB-3603] Enable more strict code checking
1d81a963840406673b0d55182825a2430b7ac385 Merge "[FAB-6716] Enabled mutual TLS support for orderer"
e90b92ba6947d84ad5a4bdad5d1004f0bfd5819f [FAB-6716] Enabled mutual TLS support for orderer
f844f860c4cc2a55f88279fca009b22662676f05 [FAB-5720] Re-submit tx if re-validation passes
28024309ad1afe464aa146aff9151cf647aac470 [FAB-6143] Cleanup dead code
2ef4823ec00934037af2f192532b5243a0538074 [FAB-6142] Move MakeChainCreationTranscation
95b95e903baab281bef7324d44120ce9d862e545 [FAB-6140] Remove the provisional config encoder
8a52d63e6b3877ae850ce47f2e7695b068172c3b Merge "[FAB-6380] fix race condition in kafka chain Halt"
978c48cd64332de3c68d5ead3d961387524568c7 [FAB-6088] Add v1.1 application capabilities flag
376c2ca28862d2c605e154d762b17db4183389fe [FAB-6485] Add capabilities hook for MSP version
e76b396602c343277476faa26ca2c7064e314795 [FAB-6380] fix race condition in kafka chain Halt
6fb5cbddee2dc179265916c3e2ff5b8cc5d4c1c4 [FAB-6509] fix error during consumption
kostas (Fri, 01 Dec 2017 15:15:56 GMT):
I can't see anything that would hinder the orderer's ability to validate/process/order messages in the list above, but I might be missing something. Please have a look and let me know.
kostas (Fri, 01 Dec 2017 15:17:21 GMT):
(At the end of it all, I think we may have to do some git-bisect.)
jyellick (Fri, 01 Dec 2017 16:36:42 GMT):
My first guess would be 0c028df008b73aa606abbe0ee06b37a4f9854210 since it introduces additional locking
kostas (Fri, 01 Dec 2017 18:00:05 GMT):
That is a very good guess. However, as our friends in Zurich informed me, the code they experimented with was no later than the week of November 13, so that would mean the first 4 commits are out of this list.
kostas (Fri, 01 Dec 2017 18:00:05 GMT):
That is a very good guess. However, as our friends in Zurich informed me after I posted this, the code they experimented with was no later than the week of November 13, so that would mean the first 4 commits are out of this list.
kostas (Fri, 01 Dec 2017 18:00:05 GMT):
That is a very good guess. However, as our friends in Zurich informed me after I posted this, the code they experimented with was published no later than the week of November 13, so that would mean the first 4 commits are out of this list.
kostas (Fri, 01 Dec 2017 18:01:59 GMT):
For context for anyone reading this: the reason we're looking into this is because our colleagues in IBM Zurich are running performance experiments. They had seen an increase in throughput performance when they switched from 1.0.x to `master` around the end of September, but noticed a regression with the code they tested around the middle of November.
kostas (Fri, 01 Dec 2017 18:01:59 GMT):
For context, for anyone reading this: the reason we're looking into this is because our colleagues in IBM Zurich are running performance experiments. They had seen an increase in throughput performance when they switched from 1.0.x to `master` around the end of September, but noticed a regression with the code they tested around the middle of November.
kostas (Fri, 01 Dec 2017 18:02:56 GMT):
I got a plan to figure this out; we'll rebase on master, and do git-bisect to figure out where the regression happens exactly.
kostas (Fri, 01 Dec 2017 18:02:56 GMT):
I think there's a fairly straightforward way of figuring this out; we'll rebase on master, and do `git-bisect` to figure out where the regression happens exactly.
jyellick (Fri, 01 Dec 2017 21:24:40 GMT):
@kostas Any inspiration on the block isolation?
kostas (Fri, 01 Dec 2017 21:28:53 GMT):
None so far, and this has been on the back of my mind all day. My work day’s not over yet (I’ll put in a couple of hours after 6pm), but I’m close to accepting defeat. I’ll post here to make it official though.
jyellick (Fri, 01 Dec 2017 21:29:35 GMT):
No rush, I am a bit burnt out myself from this lifecycle stuff, so will not plan on making any moves tonight
kostas (Sat, 02 Dec 2017 13:02:30 GMT):
Here's one last, unpolished thought that crossed my mind. We define a namespace under the configuration tree (a `Value` under `/Orderer/`?) where we enumerate `ChannelHeader` types that should be cut into their own block. This makes the coupling with the Fabric peer looser and doesn't leak info of its peer processing model. It also means we can modify the types that should be isolated later on without changing the orderer code. One obvious, big downside to this approach is that Go's lack of reflection will mean we have got go heavy on the `reflect` package to make things work. (And I'm not even sure it can work, but _if_ we agree that this suggestion has legs, we can test out its feasibility.)
kostas (Sat, 02 Dec 2017 13:02:30 GMT):
Here's one last thought. We define a namespace under the configuration tree (a `Value` under `/Orderer/`?) where we enumerate `ChannelHeader` types that should be cut into their own block. This makes the coupling with the Fabric peer looser and doesn't leak info of its peer processing model. It also means we can modify the types that should be isolated later on without changing the orderer code. One obvious, big downside to this approach is that Go's lack of reflection will mean we have got go heavy on the `reflect` package to make things work. (And I'm not even sure it can work, but _if_ we agree that this suggestion has legs, we can test out its feasibility.)
kostas (Sat, 02 Dec 2017 13:02:30 GMT):
Here's one last thought. We define a namespace under the configuration tree (a `Value` under `/Orderer/`?) where we enumerate `ChannelHeader` types that should be cut into their own block. This makes the coupling with the Fabric peer looser and doesn't leak info of its peer processing model. It also means we can modify the types that should be isolated later on without changing the orderer code. One obvious, big downside to this approach is that Go's lack of reflection will mean we have got go heavy on the `reflect` package to make things work. (And I'm not even sure it can work, but _if_ we agree that this suggestion has legs, we can evaluate its feasibility.)
simcan (Mon, 04 Dec 2017 10:29:17 GMT):
Has joined the channel.
jyellick (Mon, 04 Dec 2017 16:37:51 GMT):
So, I don't think that reflection is particularly a problem here
jyellick (Mon, 04 Dec 2017 16:38:11 GMT):
And, I would actually enumerate the types under `/Channel/Application` if it is information specified by the application to the orderer
jyellick (Mon, 04 Dec 2017 16:38:22 GMT):
The biggest problem with this approach is actually the upgrade
jyellick (Mon, 04 Dec 2017 16:38:57 GMT):
v1.0 channel configs will not define this type, but when upgraded for v1.1, to support the resources config, they would need to be there.
jyellick (Mon, 04 Dec 2017 16:39:50 GMT):
I'm not saying it's impossible, we could add custom tx processing logic which adds the new value on upgrade to v1.1 capabilities (as we do already with respect to fixing the broken mod_policy), but it's a fair bit of additional complexity which I am not entirely sure I am sold on.
jyellick (Mon, 04 Dec 2017 16:39:56 GMT):
What do you think @kostas ?
kostas (Mon, 04 Dec 2017 16:44:32 GMT):
Just to be clear: are you saying that the custom logic that will add the value when upgrading to v1.1 is a fair bit of additional complexity?
kostas (Mon, 04 Dec 2017 16:45:12 GMT):
Or getting this framework to work in general would be more complex than necessary?
jyellick (Mon, 04 Dec 2017 16:46:41 GMT):
The first -- custom logic on upgrade is, generally throwaway code, but code which must be maintained until that upgrade scenario is no longer supported. In general, I prefer to make any upgrade code as simple and small in scope as possible. As I say, it can be done though.
kostas (Mon, 04 Dec 2017 16:56:26 GMT):
Understood. I want to suggest we go for it then. However, I know you share my concerns about leaking info from the peer to the orderer. So, if I'm underestimating the amount of work involved in this, and you want to go with the original plan, I won't block it even if I disagree.
kostas (Mon, 04 Dec 2017 16:56:26 GMT):
Understood. I want to suggest we go for it then. However, I know you share my concerns about leaking info from the peer to the orderer. So, if I'm underestimating the amount of work involved in this, and you want to go with the original plan, I won't block it even though I disagree.
jyellick (Mon, 04 Dec 2017 16:59:23 GMT):
I'm fine with this, and I actually might piggy-back a bit onto this. I would like to define some additional default policies, such as `/Channel/Application/{Any,Majoity,All}OrgsPeers`, `/Channel/Application/{Any,Majority,All}OrgAdmins`. If we are already adding a new config value, could also add these in as well
jyellick (Mon, 04 Dec 2017 16:59:23 GMT):
I'm fine with this, and I actually might piggy-back a bit onto this. I would like to define some additional default policies, such as `/Channel/Application/{Any,Majoity,All}OrgsPeers`, `/Channel/Application/{Any,Majority,All}OrgsAdmins`. If we are already adding a new config value, could also add these in as well
jyellick (Mon, 04 Dec 2017 16:59:23 GMT):
I'm fine with this, and I actually might piggy-back a bit onto this. I would like to define some additional default policies, such as `/Channel/Application/{Any,Majority,All}OrgsPeers`, `/Channel/Application/{Any,Majority,All}OrgsAdmins`. If we are already adding a new config value, could also add these in as well
kostas (Mon, 04 Dec 2017 16:59:58 GMT):
Ah, perfect. I remember you talking about this last Tuesday.
jyellick (Mon, 04 Dec 2017 17:07:45 GMT):
Would you mind updating https://jira.hyperledger.org/browse/FAB-6233 with your proposal?
kostas (Mon, 04 Dec 2017 18:04:55 GMT):
(Done.)
kostas (Tue, 05 Dec 2017 20:30:01 GMT):
Do we know of a way to do `inline code` in a JIRA comment?
jyellick (Tue, 05 Dec 2017 20:30:24 GMT):
`{{inline code}}`
kostas (Tue, 05 Dec 2017 20:30:35 GMT):
You are a gentleman and a scholar, thanks.
kostas (Tue, 05 Dec 2017 20:30:46 GMT):
Very intuitive syntax, I'd add.
jyellick (Tue, 05 Dec 2017 20:31:17 GMT):
I wish it did slack/RC style background coloring, but it is at least monospace.
kostas (Tue, 05 Dec 2017 20:34:14 GMT):
I don't like the fact that if you use the "Visual" interface, you need to apply the markup in a certain way (so that their JavaScript catches it and transforms it on the fly), otherwise your transformation won't work. For instance, if you do ``{{code}}`` but apply the end brackets first, you'll literally end up with ``code`` instead of `code`.
kostas (Tue, 05 Dec 2017 20:34:14 GMT):
I don't like the fact that if you use the "Visual" interface, you need to apply the markup in a certain way (so that their JavaScript catches it and transforms it on the fly), otherwise your transformation won't work. For instance, if you do `{{code}}` but apply the end brackets first, you'll literally end up with ``code`` instead of `code`.
kostas (Tue, 05 Dec 2017 20:34:14 GMT):
I don't like the fact that if you use the "Visual" interface, you need to apply the markup in a certain way (so that their JavaScript catches it and transforms it on the fly), otherwise your transformation won't work. For instance, if you do `{{code}}` but apply the end brackets first, you'll literally end up with {{code}} instead of `code`.
jyellick (Tue, 05 Dec 2017 20:35:56 GMT):
Quite right, I find myself sticking to the non-visual interface for composition, then flip over for proof reading
kostas (Tue, 05 Dec 2017 20:36:22 GMT):
That's a good one, perhaps I should adopt this strategy as well.
kostas (Tue, 05 Dec 2017 22:03:52 GMT):
Don't know if I've posted about this here, but http://glogg.bonnefon.org/ is pretty amazing when it comes to analyzing logs.
kostas (Tue, 05 Dec 2017 22:03:52 GMT):
Don't know if I've posted about this here before, but http://glogg.bonnefon.org/ is pretty amazing when it comes to analyzing logs.
wanghhao (Thu, 07 Dec 2017 09:11:39 GMT):
Has joined the channel.
JayJong (Fri, 08 Dec 2017 11:12:06 GMT):
Has joined the channel.
guoger (Sat, 09 Dec 2017 03:59:22 GMT):
could somebody remind me again what `MSP` policy is for?
```
message Policy {
enum PolicyType {
UNKNOWN = 0; // Reserved to check for proper initialization
SIGNATURE = 1;
MSP = 2;
IMPLICIT_META = 3;
}
int32 type = 1; // For outside implementors, consider the first 1000 types reserved, otherwise one of PolicyType
bytes value = 2;
}
```
jyellick (Sun, 10 Dec 2017 04:03:41 GMT):
@guoger It's unimplemented -- it's something that the crypto folks had requested, but ultimately the SIGNATURE policy was sufficient
jyellick (Sun, 10 Dec 2017 04:03:41 GMT):
@guoger It's unimplemented -- it's something that the crypto folks had requested, but ultimately the `SIGNATURE` policy was sufficient
guolidong (Tue, 12 Dec 2017 06:03:16 GMT):
Has joined the channel.
guoger (Tue, 12 Dec 2017 07:22:45 GMT):
Somebody's asked me about pruning of ledger. I wonder what's the status? do we have a design doc of this? thx. cc @sanchezl
sanchezl (Tue, 12 Dec 2017 13:05:50 GMT):
I will not have an update on pruning design until January.
kostas (Tue, 12 Dec 2017 19:07:28 GMT):
Can we kick this process off in its respective JIRA sometime this week? Even if it's just [a rough and incomplete draft](https://medium.com/@ienjoy/mcdonalds-theory-9216e1c9da7d) at the beginning, it is better than nothing, and it will allow others to chime in and contribute as well.
guoger (Wed, 13 Dec 2017 01:38:53 GMT):
@kostas very interesting blog...
guoger (Wed, 13 Dec 2017 03:21:34 GMT):
Anybody got the same issue here? https://gerrit.hyperledger.org/r/#/c/16045/
> LICENSE file in this commit gives me line ending problems with git on OSX. basically i checked out latest master and this file is stubbornly stuck at "changed but not staged"
jyellick (Wed, 13 Dec 2017 03:22:02 GMT):
Ah, this seems to be a Mac git problem
jyellick (Wed, 13 Dec 2017 03:22:15 GMT):
There is someone complaining about it in #fabric
jyellick (Wed, 13 Dec 2017 03:22:37 GMT):
He's pinged me directly as well, if you could post to #fabric to confirm the same problem, would appreciate it
guoger (Wed, 13 Dec 2017 03:39:55 GMT):
Will Lathi also confirmed same issue in that CR comments. I've suggested @manu to re-commit that file with proper line ending in #fabric
manu (Wed, 13 Dec 2017 03:39:56 GMT):
Has joined the channel.
wanghhao (Mon, 18 Dec 2017 10:06:23 GMT):
Has left the channel.
zhishui (Tue, 19 Dec 2017 08:12:04 GMT):
Has joined the channel.
kostas (Wed, 20 Dec 2017 00:05:22 GMT):
I am looking over the upgrade doc and realize that something is not 100% clear to me regarding the `SetChannelModPolicyDuringCreate` fix.
kostas (Wed, 20 Dec 2017 00:22:27 GMT):
Namely, the function's name and its description imply that it takes effect only during channel creation requests.
kostas (Wed, 20 Dec 2017 00:24:44 GMT):
However given that you also want to have this fix so that you can do edits on the `/Channel` level, I am guessing that this is not only the case?
kostas (Wed, 20 Dec 2017 00:25:55 GMT):
The trigger for this observation is the 2-step approach Jeff has adopted in his `upgrade.feature`, see: https://github.com/jeffgarratt/fabric-prototype/blob/master/features/upgrade.feature#L523
kostas (Wed, 20 Dec 2017 00:33:26 GMT):
I am looking at the references to the `NewChannelConfig` method (which is the only method referencing `SetChannelModPolicyDuringCreate`) and both seem to apply only during the creation of a new channel. I suspect I may be misunderstanding something, particularly w/r/t the use of templates during configuration updates.
kostas (Wed, 20 Dec 2017 00:33:26 GMT):
I am looking at the references to the `NewChannelConfig` method (which is the only method referencing `SetChannelModPolicyDuringCreate`) and both seem to apply only during the creation of a new channel. I suspect I may be missing something, particularly w/r/t the use of templates during configuration updates.
jyellick (Wed, 20 Dec 2017 15:19:01 GMT):
@kostas There are two (maybe three) pieces to this puzzle.
First, there were two ways that your channel could be broken.
1. You created the channel create tx with a version of `configxgen` prior to 1.0.1.
2. You created the channel with a version of the `orderer`prior to 1.1.0 (or without the capability enabled)
For (1), this is a tool and it has been fixed, no non-determinism possible. For (2), since the channel creation is validated by each orderer, they must do this consistently, this is why there is the check for `SetChannelModPolicyDuringCreate()`.
So, with (1) and (2) fixed, you can no longer create broken channels.
However, none of this does anything to address the already broken channels. So, this is where that custom upgrade process you described comes in.
kostas (Wed, 20 Dec 2017 15:59:18 GMT):
@jyellick: This is the bit I'm still not clear on.
kostas (Wed, 20 Dec 2017 15:59:18 GMT):
@jyellick: This is the part I'm still not clear on.
kostas (Wed, 20 Dec 2017 15:59:19 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=Kwdid7JJgs732XmJb
kostas (Wed, 20 Dec 2017 15:59:47 GMT):
Or to put differently:
kostas (Wed, 20 Dec 2017 15:59:51 GMT):
> So, this is where that custom upgrade process you described comes in.
jyellick (Wed, 20 Dec 2017 16:00:19 GMT):
`SetChannelModPolicyDuringCreate()` is only related to channel creation, you are correct, it has nothing to do with the 'custom upgrade'
jyellick (Wed, 20 Dec 2017 16:00:43 GMT):
There is no method exposed with respect to the custom upgrade, because this takes place within the configtx processing itself.
jyellick (Wed, 20 Dec 2017 16:01:13 GMT):
https://github.com/hyperledger/fabric/blob/master/common/configtx/configmap.go#L159-L187
kostas (Wed, 20 Dec 2017 16:03:16 GMT):
Great, that's the piece that I was missing. Thank you.
kostas (Wed, 20 Dec 2017 16:04:29 GMT):
So, this means that:
kostas (Wed, 20 Dec 2017 16:06:03 GMT):
When you push a configuration update that creates a capabilities map under the Orderer config group, you will come up with a configuration that has all of its empty `mod_policy` fields set.
kostas (Wed, 20 Dec 2017 16:06:20 GMT):
True, false?
kostas (Wed, 20 Dec 2017 16:06:37 GMT):
I suspect there may be subtle cases that I'm missing with this statement.
jyellick (Wed, 20 Dec 2017 16:06:52 GMT):
True
jyellick (Wed, 20 Dec 2017 16:09:11 GMT):
Because this is an admittedly hacky one-off which risks non-determinism, the rule was intentionally kept as simple as that.
jyellick (Wed, 03 Jan 2018 16:03:09 GMT):
@sanchezl @guoger Would one of you have a chance to take a look at [ ](https://chat.hyperledger.org/channel/fabric-orderer?msg=7gW3kqzw2syNSu2bh) ? My gut says that Yacov is right and we should set up the auto-purging
jyellick (Wed, 03 Jan 2018 16:03:09 GMT):
@sanchezl @guoger Would one of you have a chance to take a look at [ ](https://chat.hyperledger.org/channel/fabric-orderer?msg=7gW3kqzw2syNSu2bh) ? My gut says that @yacovm is right and we should set up the auto-purging
jyellick (Wed, 03 Jan 2018 16:03:09 GMT):
@sanchezl @guoger Would one of you have a chance to take a look at [ ](https://chat.hyperledger.org/channel/fabric-orderer?msg=7gW3kqzw2syNSu2bh) the below? My gut says that @yacovm is right and we should set up the auto-purging
yacovm (Wed, 03 Jan 2018 16:08:31 GMT):
Thanks Jason
gurel (Wed, 03 Jan 2018 17:46:50 GMT):
Has joined the channel.
guoger (Thu, 04 Jan 2018 01:34:27 GMT):
@jyellick I'll take a look today
guoger (Thu, 04 Jan 2018 06:01:56 GMT):
@jyellick @yacovm reading your request and I feel it's more of a problem for zk admins? should it be our responsibility to provide a recommended zk configuration? nevertheless, i suppose if we want to do this, it should be part of ZK dockerfile, like `ZK_AUTOPURGE_PURGEINTERVAL=1` and `ZK_AUTOPURGE_SNAPRETAINCOUNT=3`.
jyellick (Thu, 04 Jan 2018 06:03:00 GMT):
Is there any downside to including those parameters?
guoger (Thu, 04 Jan 2018 06:04:17 GMT):
not really.. however, if it's a busy zk cluster, admin shouldn't clear logs during peak hours as it affects performance significantly
guoger (Thu, 04 Jan 2018 06:04:17 GMT):
not really.. however, if it's a busy zk cluster, admin shouldn't clear logs during peak hours as it affects performance significantly. so a cron job is actually better
guoger (Thu, 04 Jan 2018 06:05:17 GMT):
my opinion on this is to leave everything to default, like what we are doing today
guoger (Thu, 04 Jan 2018 06:05:59 GMT):
and in production deployment, operators should follow zk admin guide
yacovm (Thu, 04 Jan 2018 06:06:54 GMT):
There are no admins
yacovm (Thu, 04 Jan 2018 06:07:02 GMT):
In the docler images we provide
yacovm (Thu, 04 Jan 2018 06:07:14 GMT):
Thats how i see that
jyellick (Thu, 04 Jan 2018 06:07:15 GMT):
Would like to hear from @kostas, for them images we provide, I would lean towards low maintenance over the performance. Certainly anyone in production should employ a qualified Kafka/ZK admin
jyellick (Thu, 04 Jan 2018 06:07:15 GMT):
Would like to hear from @kostas, but for the images we provide, I would lean towards low maintenance over the performance. Certainly anyone in production should employ a qualified Kafka/ZK admin
jyellick (Thu, 04 Jan 2018 06:07:15 GMT):
Would like to hear from @kostas, but for the images we provide, I would lean towards low maintenance over performance. Certainly anyone in production should employ a qualified Kafka/ZK admin
yacovm (Thu, 04 Jan 2018 06:07:55 GMT):
We provide images. If the org has admins then they would provode theor own zookeeper
guoger (Thu, 04 Jan 2018 06:08:45 GMT):
So the goal is to provide production-ready ZK images?
yacovm (Thu, 04 Jan 2018 06:08:50 GMT):
Also, performance? I thought zookeeper is used to elect a leader. Is it in the transaction critical path of kafka?
guoger (Thu, 04 Jan 2018 06:09:20 GMT):
oh, I meant performance of ZK, not kafka
guoger (Thu, 04 Jan 2018 06:09:20 GMT):
oh, I meant performance of ZK, not kafka, that's why I said it shouldn't affect us much
yacovm (Thu, 04 Jan 2018 06:09:43 GMT):
The goal is to provide images that dont make the users disk fill up :wink:
yacovm (Thu, 04 Jan 2018 06:10:14 GMT):
So it will not come here crying that kafka OSN is down, and he has no idea what to do
yacovm (Thu, 04 Jan 2018 06:11:56 GMT):
LOL @ @jyellick "qualified kafka/ZK admim :joy:
yacovm (Thu, 04 Jan 2018 06:14:30 GMT):
If we do this change then a qualified fabric admin doesnt need to be also a qualified ZK admin on top of other hats
yacovm (Thu, 04 Jan 2018 06:14:30 GMT):
If we do this change then a qualified fabric admin doesnt need to be also a qualified ZK admin
guoger (Thu, 04 Jan 2018 06:16:35 GMT):
LOL. alright, I'm not strongly against this, so if we've reached a consensus here, I'll submit a patch for it. cc @kostas
yacovm (Thu, 04 Jan 2018 06:17:21 GMT):
Would also like to hear @mastersingh24 s opinion
mastersingh24 (Thu, 04 Jan 2018 06:17:21 GMT):
Has joined the channel.
guoger (Thu, 04 Jan 2018 06:19:51 GMT):
btw, @yacovm are you addressing this? https://chat.hyperledger.org/channel/fabric-orderer?msg=9pcGebhCaBrP596Ym
yacovm (Thu, 04 Jan 2018 06:30:55 GMT):
Yes precisely
yacovm (Thu, 04 Jan 2018 06:31:05 GMT):
@guoger
kostas (Thu, 04 Jan 2018 15:00:25 GMT):
In for getting the `ZK_AUTOPURGE_*` settings in.
guoger (Fri, 05 Jan 2018 05:33:54 GMT):
https://gerrit.hyperledger.org/r/#/c/16589/ -> https://jira.hyperledger.org/browse/FAB-7609
cc @yacovm @kostas @jyellick
guoger (Mon, 08 Jan 2018 08:50:33 GMT):
@kostas @jyellick @sanchezl submitted a patch to print config parameters during boot time for orderer: https://gerrit.hyperledger.org/r/#/c/16653/
kostas (Mon, 08 Jan 2018 14:22:48 GMT):
@guoger: Thanks, will look into it. For context to the others, this seemed like a useful thing to add after discussing it with Scott Z and team.
kostas (Wed, 10 Jan 2018 15:48:56 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=f3Fdmw8YdSyTyH37z
kostas (Wed, 10 Jan 2018 15:51:00 GMT):
For anyone curious about this, turns out that a couple of months ago I had set an `alias` for `protoc` in my `.bashrc` that would expand `protoc` automatically to `protoc --go_out==plugins=grpc:.`. It took a couple of hours of uninstalling and reinstalling everything go-related, running `dtrace`, etc. to figure this out.
kostas (Wed, 10 Jan 2018 15:51:00 GMT):
For anyone curious about this, turns out that a couple of months ago I had set an `alias` for `protoc` in my `.bashrc` that would expand `protoc` automatically to `protoc --go_out==plugins=grpc:.`. It took a couple of hours of uninstalling and reinstalling everything go-related, running `dtrace`, etc. to figure this out. Lovely.
jyellick (Wed, 10 Jan 2018 16:48:34 GMT):
Ouch. Glad you got it figure out.
jyellick (Wed, 10 Jan 2018 22:18:47 GMT):
Is JIRA/Gerrit slow/generally broken for anyone else?
guoger (Thu, 11 Jan 2018 01:33:10 GMT):
@jyellick it works fine for me
guoger (Thu, 11 Jan 2018 03:24:42 GMT):
When kafka log retention time expires, and *all* the remaining messages in kafka are *connect msg* (restart orderer several times without new tx), orderer would fail to connect to kafka due to offset out of range. Do we consider this as a known issue and will be solved by log pruning?
guoger (Thu, 11 Jan 2018 03:24:49 GMT):
cc @jyellick @kostas
guoger (Thu, 11 Jan 2018 03:24:49 GMT):
cc @jyellick @kostas @sanchezl
jyellick (Thu, 11 Jan 2018 03:30:24 GMT):
I know @sanchezl has looked into this more, and I am struggling to remember his findings. I thought there was a way to configure both a minimum number of messages retained, as well as a time period?
jyellick (Thu, 11 Jan 2018 03:30:55 GMT):
Assuming the minimum is set to be reasonably high (say, a few thousand?) it would require many of the connect messages, but I suppose it could still potentially be a problem
jyellick (Thu, 11 Jan 2018 03:31:38 GMT):
No quick fix jumps to mind though, unless there is a new block written, we have no where to write the metadata, so this could eventually happen (especially say after many years on an idle channel)
sanchezl (Thu, 11 Jan 2018 03:39:13 GMT):
I don't think we've explicitly thought of this idle channel scenario. In agreement with @jyellick, this is a scenario that *can* happen. I'll investigate further.
guoger (Thu, 11 Jan 2018 03:40:11 GMT):
which is the min retention size config? `log.retention.bytes`?
sanchezl (Thu, 11 Jan 2018 03:41:50 GMT):
yes
guoger (Thu, 11 Jan 2018 03:46:04 GMT):
@sanchezl trying to understand the logic here. with this option set, it would keep at least this much bytes of log, even they've existed longer than retention time?
sanchezl (Thu, 11 Jan 2018 04:04:14 GMT):
Bear with me, I might remember this this wrong the first time:
• when a log segment file reaches size `log.segment.bytes`, a new log segment file is created and the previous log segment is _closed_. Only closed segments can be expired/deleted.
• *Before 0.10.1*, the retention due to on `log.retention.ms` is based on the last modified time of the log segment. *After 0.10.1*, the retention time is based on the timestamp of the last message in the segment.
• closed segments can then expire when either `log.retention.bytes` or `log.retention/ms` criteria is met.
sanchezl (Thu, 11 Jan 2018 04:21:32 GMT):
A connect message should be about 20bytes plus the bytes needed to encoded the topic name. I'm not sure what the rate of CONNECT messages is, but it would take over 30 million of them to fill a 1GB segment completely.
jyellick (Thu, 11 Jan 2018 04:22:57 GMT):
> closed segments can then expire when either `log.retention.bytes` *or* `log.retention/ms` criteria is met.
Did you mean _or_ here or _and_ ?
jyellick (Thu, 11 Jan 2018 04:22:57 GMT):
> closed segments can then expire when either `log.retention.bytes` _or_ `log.retention/ms` criteria is met.
Did you mean _or_ here or _and_ ?
jyellick (Thu, 11 Jan 2018 04:22:57 GMT):
> closed segments can then expire when either `log.retention.bytes` _or_ `log.retention/ms` criteria is met.
Did you mean *or* here or *and* ?
guoger (Thu, 11 Jan 2018 04:50:40 GMT):
> Only closed segments can be expired/deleted.
hmm.. `log.segment.bytes` is 1073741824 by default. as you said, it should be very hard to fill it with connect msg. However, i tried e2e example with `log.retention.minutes=1`, old message *are* discarded when it expires
jyellick (Thu, 11 Jan 2018 04:59:49 GMT):
What is `log.retention.bytes` set to for your test
jyellick (Thu, 11 Jan 2018 04:59:49 GMT):
What is `log.retention.bytes` set to for your test?
guoger (Thu, 11 Jan 2018 05:03:20 GMT):
1) left as default (-1), the problem described before occurred
2) set to 1000000, problem went away
guoger (Thu, 11 Jan 2018 05:10:48 GMT):
actually no... the problem is still there
guoger (Thu, 11 Jan 2018 05:11:14 GMT):
it seems that `log.retention.bytes` doesn't prevent logs being deleted
jyellick (Thu, 11 Jan 2018 05:11:19 GMT):
What if you set `log.retention.bytes` > `log.segment.bytes` ?
guoger (Thu, 11 Jan 2018 05:14:13 GMT):
trying
guoger (Thu, 11 Jan 2018 05:23:21 GMT):
still, old logs are deleted, even though the size is way below `log.retention.bytes`
jyellick (Thu, 11 Jan 2018 05:24:16 GMT):
That's unfortunate -- it sounds like it is indeed an *or* type relationship between those two settings. *and* seems like it would be superior
guoger (Thu, 11 Jan 2018 05:26:29 GMT):
yeah... `log.retention.bytes` doesn't guarantee minimum bytes of logs to be retained
guoger (Thu, 11 Jan 2018 05:27:26 GMT):
I guess the only way to solve this problem is to set a reasonably large `log.retention.(time)`
guoger (Thu, 11 Jan 2018 05:29:09 GMT):
but again I'm a bit confused. `log.roll.hours` is set to 168 hours by default. Why new log files are still created fairly quickly?
guoger (Thu, 11 Jan 2018 05:29:25 GMT):
is it affected by `log.retention.(time)`?
jyellick (Thu, 11 Jan 2018 05:30:41 GMT):
@sancezl I think you've done more research on this than anyone?
jyellick (Thu, 11 Jan 2018 05:30:41 GMT):
@sanchezl I think you've done more research on this than anyone?
guoger (Thu, 11 Jan 2018 06:26:43 GMT):
it seems that `log.retention.(time)` does affect the frequency of kaka log rolling
guoger (Thu, 11 Jan 2018 06:35:14 GMT):
but it's still weird that `log.roll.ms` is not really respected even if `log.retention.(time)` is way greater than it.
guoger (Thu, 11 Jan 2018 07:12:43 GMT):
ah... kafka only checks these criteria when new msg comes in...
guoger (Thu, 11 Jan 2018 08:07:03 GMT):
to summarize:
- log retention in Kafka is *scheduled* job, specified by `log.retention.check.interval`. If `log.retention.(time)` expires *or* `log.retention.bytes` satisfied, logs are deleted.
- if `log.retention.(time)` expires, it forces log rolling, regardless of `log.segment.ms`
- `log.retention.bytes` does *not* force log rolling, and it's checked against *closed* log segments
- `log.roll.(time)` is checked only when new message comes into kafka
guoger (Thu, 11 Jan 2018 08:08:36 GMT):
Therefore, we don't really have an ideal solution for original problem but setting `log.retention.(time)` reasonably large (default 168 hrs should be good)
> When kafka log retention time expires, and *all* the remaining messages in kafka are *connect msg* (restart orderer several times without new tx), orderer would fail to connect to kafka due to offset out of range. Do we consider this as a known issue and will be solved by log pruning?
kostas (Thu, 11 Jan 2018 15:53:01 GMT):
Late to the party here.
kostas (Thu, 11 Jan 2018 15:53:08 GMT):
A couple of notes.
kostas (Thu, 11 Jan 2018 15:53:16 GMT):
For whatever it's worth.
kostas (Thu, 11 Jan 2018 15:53:38 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=wMcY762EwL7tYYajs
kostas (Thu, 11 Jan 2018 15:53:58 GMT):
As Luis wrote, it is indeed an OR relationship:
kostas (Thu, 11 Jan 2018 15:53:58 GMT):
As Luis wrote, it is indeed an OR relationship. From the Kafka Definitive Guide book:
kostas (Thu, 11 Jan 2018 15:54:25 GMT):
> If you have specified a value for both `log.retention.bytes` and `log.retention.ms`, messages may be removed when either criteria is met.
kostas (Thu, 11 Jan 2018 15:55:12 GMT):
I am slightly lost as for the rest of the discussion.
kostas (Thu, 11 Jan 2018 15:55:26 GMT):
What does `log.retention.*` have to do with log rolling?
kostas (Thu, 11 Jan 2018 15:55:57 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=XQSMknvmFRG3SZqmL
kostas (Thu, 11 Jan 2018 15:59:08 GMT):
So this is an issue, though as Luis noted the chances of this one happening are tiny. If we were even more paranoid, we could enable log compaction where a connect message is posted under, say, `key=CONNECT`, and all other messages are posted under `key=GUID`. The only real risk is if you get two colliding GUIDs within the channel, but I would probably take those odds.
kostas (Thu, 11 Jan 2018 15:59:49 GMT):
With log compaction enabled, you ensure that only one connect message is present. The tradeoff being, GUID collision for all other messages.
kostas (Thu, 11 Jan 2018 16:01:15 GMT):
Perhaps there is an even simpler approach where you add logic to the orderer and (a) attempt to read the partition first without posting a connect message, (b) recover the panic that will occur if the partition is empty, and (c) only then post the connect message.
guoger (Fri, 12 Jan 2018 02:23:02 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=Jr8iQcrHv2xZWMvsf) @kostas `log.retention.(time)` would force log rolling, but `log.retention.bytes` wouldn't.
guoger (Fri, 12 Jan 2018 02:23:27 GMT):
But this is just a behavior probably missing in Kafka docs
guoger (Fri, 12 Jan 2018 04:48:12 GMT):
I'll file a jira and try to submit a patch for this. It actually happened in requester's semi-production env, where their channel is idle for 168 hours during new year holiday, rebooted several times, and orderer failed to reconnect
guoger (Fri, 12 Jan 2018 04:48:37 GMT):
I'm inclined to go for the 'simpler' solution
guoger (Fri, 12 Jan 2018 10:54:39 GMT):
filed https://jira.hyperledger.org/browse/FAB-7707
kostas (Fri, 12 Jan 2018 13:29:45 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=EanYmDGrs8FkTio83
kostas (Fri, 12 Jan 2018 13:29:56 GMT):
Ah, I read your JIRA description and it makes sense now.
kostas (Fri, 12 Jan 2018 13:30:03 GMT):
Thanks for filing this.
guoger (Thu, 18 Jan 2018 06:45:03 GMT):
IIUC, system channel name is hardcoded to be `testchainid`. Shouldn't we rename it to be something more clear, i.e. `systemchannel`?
jyellick (Thu, 18 Jan 2018 14:33:03 GMT):
So, it is not hard-coded, if you generate your genesis block for bootstrapping, you may pick any name you wish
jyellick (Thu, 18 Jan 2018 14:33:15 GMT):
I would say more accurately, it is defaulted to testchainid
zhoui13 (Fri, 19 Jan 2018 02:06:35 GMT):
Has joined the channel.
jyellick (Fri, 19 Jan 2018 21:39:36 GMT):
@sanchezl @guoger @kostas
We updated the version of Kafka in `orderer.yaml` to be 0.10.0.2. This can be seen to be causing a few problems:
https://jira.hyperledger.org/browse/FAB-7825
https://jira.hyperledger.org/browse/FAB-7739
and I solved this for an internal team recently as well. I just wanted to ask if we wanted to reconsider modifying this default or not. I would argue for a true upgrade, users should update their binary, and not their config file, in which case nothing breaks. Still, I thought it was worth raising for discussion here.
sanchezl (Fri, 19 Jan 2018 21:46:16 GMT):
I've gone back and forth on this one myself, one way or another someone it going to be inconvenienced. If they are using our docker images, then they are updating more than just the binary, as the image contains an updated orderer.yml within.
jyellick (Fri, 19 Jan 2018 21:46:57 GMT):
What are the advantages to specifying the newer wire format?
sanchezl (Fri, 19 Jan 2018 21:47:28 GMT):
We have updated the docs regarding this: http://hyperledger-fabric.readthedocs.io/en/latest/kafka.html#kafka-protocol-version-compatibility
sanchezl (Fri, 19 Jan 2018 21:48:16 GMT):
and the "internal" upgrade docs : https://docs.google.com/document/d/16ZogU_InYeLZZ8lTGNdBhd4Iosi7T4KP3ppFSEUcI30/edit?usp=sharing explicitly mention the need to update the docker compose files
sanchezl (Fri, 19 Jan 2018 21:49:29 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=RXzC28Syk375eCqAw) @jyellick There is a performance hit when using the older wire format as kafka has to translate between the current format and the client's format (Kafka uses the same format for on-disk-storage).
sanchezl (Fri, 19 Jan 2018 21:50:56 GMT):
Once you go to Kafka 0.11+, the newer formats enable a new scheme designed to prevent the log truncation errors that can occur after abrupt shutdowns.
sanchezl (Fri, 19 Jan 2018 22:14:08 GMT):
If anyone is interested, here is the performance report published by Kafka:
https://docs.google.com/spreadsheets/d/1dHY6M7qCiX-NFvsgvaE0YoVdNq26uA8608XIh_DUpI4/edit#gid=61107630
• Column D: Producer delta.
• Column I: Consumer delta.
• Column L: Disk size delta.
sanchezl (Fri, 19 Jan 2018 22:14:08 GMT):
If anyone is interested, here is the performance report published by Kafka:
https://docs.google.com/spreadsheets/d/1dHY6M7qCiX-NFvsgvaE0YoVdNq26uA8608XIh_DUpI4/edit#gid=61107630
• Column D : Producer delta.
• Column I: Consumer delta.
• Column L: Disk size delta.
jyellick (Fri, 19 Jan 2018 22:25:23 GMT):
Seems like if we are going to break everyone going to v0.10.0.2 we should just go to v0.11+?
jyellick (Fri, 19 Jan 2018 22:25:23 GMT):
Seems like if we are going to break everyone (who doesn't follow instructions) going to v0.10.0.2 we should just go to v0.11+?
Brucepark (Sat, 20 Jan 2018 06:15:22 GMT):
Has joined the channel.
kostas (Mon, 22 Jan 2018 13:42:01 GMT):
I'm missing something here, and cannot follow the recommendations provided. Can anyone recap?
kostas (Mon, 22 Jan 2018 13:49:12 GMT):
Based on what I see, upgrading the default to 0.11 as Jason suggests seems like the right move here?
kostas (Mon, 22 Jan 2018 14:15:14 GMT):
I'd also note that the `orderer.yaml` says `0.10.2.1` while our documentation says `0.10.2.0`.
jyellick (Mon, 22 Jan 2018 20:42:46 GMT):
@sanchezl @guoger What do you guys think? If we intend to change the version of Kafka in our docker files, I would think this should be done pre-alpha (so, stat)
sanchezl (Mon, 22 Jan 2018 20:51:19 GMT):
If we reset the default to `0.9.0.1`, this would satisfy the _update the binary without updating the config_ "requirement"(?). (also note that 'binary' for some is the docker image, which contains a a new `orderer.yaml`). While I am worried about performance implications: 1. we haven't done the performance testing yet, and 2, I think people will more easily swallow having to 'speed up' the default with the cool new setting, vs. having to deal with upgrades that require a change in config.
sanchezl (Mon, 22 Jan 2018 20:58:05 GMT):
I can take over one of the defects mentioned earlier to:
• Revert the the default `Kafka.Version` to `0.9.0.1` in `orderer.yaml` and the `orderer`binary,.
• Keep the actual Kafka version bundled in our images as is (v0.10.2.1). If we were to move up again, I would shoot for `v1.0.0`.
I think this would save us trouble in the short run. In the future we can revisit. Kafka has a version/capability discovery API, but we need to drop support for 0.9.0.1 in order to use it properly.
jyellick (Mon, 22 Jan 2018 20:59:21 GMT):
> Kafka has a version/capability discovery API
I was surprised that this was not present already
jyellick (Mon, 22 Jan 2018 20:59:53 GMT):
Should we not update our docker images to v1.0.0 (even if we leave `orderer.yaml` as is?)
jyellick (Mon, 22 Jan 2018 20:59:53 GMT):
Should we not update our docker images to v1.0.0 (even if we make `orderer.yaml` reference 0.9.0.1 again?)
sanchezl (Mon, 22 Jan 2018 21:09:37 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=QtdDc2xTi2Hq8HRBx) @jyellick
Most of the testing has been done on the older versions. The testers are going to make sure to run alpha on v0.11 and v1.0.0. That should give us some more confidence if we then decide to update.
jyellick (Mon, 22 Jan 2018 21:17:51 GMT):
Are you concerned that we will see problems on kafka v1.0.0?
sanchezl (Mon, 22 Jan 2018 21:22:06 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=YJrmhkgyHSkRvSiz2) @jyellick I'm not as concerned as I was when I last considered it.
jyellick (Mon, 22 Jan 2018 21:23:14 GMT):
In general, for the least pain for our users, I would think we would want to minimize the number of times we change the Kafka version. If we can confidently 'ship' Kafka v1.0.0 for our v1.1, then perhaps we will not have to change Kafka version for some time? Will leave the call up to you.
sanchezl (Mon, 22 Jan 2018 21:28:51 GMT):
Any thoughts on the 0.9.0.1 setting?
jyellick (Mon, 22 Jan 2018 21:31:40 GMT):
Would it be possible to simply enable the auto-negotiation of version if we went to Kafka v1.0.0?
sanchezl (Mon, 22 Jan 2018 21:58:02 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=Ta4GKAxhQvEBXvPeE) @jyellick IT's not enabled in the client we use `sarama`, mainly because of their support for older versions. I can look into calling the API directly and interpreting the results.
jyellick (Mon, 22 Jan 2018 22:12:03 GMT):
I do not think we want to get into a game of implementing workarounds for sarama, if anything, I would consider doing the work upstream, but it sounds like this is not an option for v1.1
Glen (Tue, 23 Jan 2018 07:06:59 GMT):
Has joined the channel.
Glen (Tue, 23 Jan 2018 07:10:11 GMT):
Has left the channel.
sanchezl (Wed, 24 Jan 2018 21:55:30 GMT):
Clipboard - January 24, 2018 4:55 PM
sanchezl (Wed, 24 Jan 2018 21:56:10 GMT):
https://gerrit.hyperledger.org/r/c/17155/
guoger (Thu, 25 Jan 2018 11:04:44 GMT):
@kostas for https://jira.hyperledger.org/browse/FAB-7707, I could take it.
>The hacky, pruning-enabled mode should be covered just in the FAQ, and
Do we have a faq section in doc?
>We fix log.retention.(time) to -1
going to fix this in kafka baseimage
kostas (Thu, 25 Jan 2018 14:53:57 GMT):
> Do we have a faq section in doc?
Ah, we do not. I have an FAQ doc out there as a draft changeset, just sent you an invitation.
kostas (Thu, 25 Jan 2018 14:53:57 GMT):
> Do we have a faq section in doc?
@guoger: Ah, we do not. I have an FAQ doc out there as a draft changeset, just sent you an invitation.
jyellick (Thu, 25 Jan 2018 15:03:35 GMT):
@sanchezl What is the maximum supported Kafka version which can be specified for the version of Sarama shipped in v1.0.x?
kostas (Thu, 25 Jan 2018 15:06:25 GMT):
(Officially: 0.10.2)
kostas (Thu, 25 Jan 2018 15:07:08 GMT):
https://github.com/hyperledger/fabric/blob/release/vendor/vendor.json#L35
https://github.com/Shopify/sarama/releases/tag/v1.12.0
https://github.com/Shopify/sarama/pull/867/files#diff-7d1c2a3334601b6c1958aae0a594cba8R151
jyellick (Thu, 25 Jan 2018 16:02:01 GMT):
@sanchezl Could you respond to @guoger in https://gerrit.hyperledger.org/r/c/17155/ ? Would like to get this merged pre-alpha if possible.
guoger (Fri, 26 Jan 2018 02:16:56 GMT):
https://gerrit.hyperledger.org/r/#/c/17233/ for FAB-7707 (setting `log.retention.ms=-1`)
guoger (Fri, 26 Jan 2018 07:26:02 GMT):
to vendor dependencies, are we recommended to use `godep`, `govendor` or `glide`?
kostas (Fri, 26 Jan 2018 12:31:03 GMT):
@guoger: We use govendor.
guoger (Fri, 26 Jan 2018 13:49:34 GMT):
theoretically, shouldn't `govendor list +e` always be the subset of `govendor list`?
sanchezl (Fri, 26 Jan 2018 20:53:34 GMT):
That would be my understanding. [ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=3Ltenn3moTLFCpPov) @guoger
guoger (Sat, 27 Jan 2018 01:51:34 GMT):
@sanchezl but `govendor list +e` gives me some packages that don't exist in `govendor list`... which is really weird..
sanchezl (Sat, 27 Jan 2018 02:10:55 GMT):
@guoger , yes that is strange. I just confirmed and I get all the `e` packages in `govendor list` when I run `govendor list +e`, and nothing more. I'm on `go version go1.9.2 darwin/amd64` and `govendor -version` = `v1.0.8`
SjirNijssen (Sun, 28 Jan 2018 09:42:30 GMT):
Has joined the channel.
rockleelx (Tue, 30 Jan 2018 04:13:31 GMT):
Has joined the channel.
MartinKrmer (Tue, 30 Jan 2018 13:49:32 GMT):
Has joined the channel.
Ashish (Mon, 05 Feb 2018 15:45:57 GMT):
Hi all, If i go inside the folder *"/var/hyperledger/production/orderer/chains/"* i can find my channels there. Inside the directory corresponding to my channel, I could find a file called *"blockfile_000000"* . It had all the inputs which I had passed during my Instantiation of all the chaincodes which are there in that channel. I think this file being available as a plain text file is a data risk. Isnt it ?
Can't anything be done about this ?
Ashish (Mon, 05 Feb 2018 15:45:57 GMT):
Hi all, If i go inside the folder *"/var/hyperledger/production/orderer/chains/"* of my Orderer, I can find my channels there. Inside the directory corresponding to my channel, I could find a file called *"blockfile_000000"* . It had all the inputs which I had passed during my Instantiation of all the chaincodes which are there in that channel. I think this file being available as a plain text file is a data risk. Isnt it ?
Can't anything be done about this ?
jyellick (Mon, 05 Feb 2018 18:16:35 GMT):
@Ashish I'm not sure what you would suggest? If it were encrypted, the process would need the key to decrypt it. I would think you would want a data protection solution at a different layer, such as LUKS or similar.
Ashish (Tue, 06 Feb 2018 05:29:14 GMT):
@jyellick I got the point from the Orderer development point of view. And I am trying to employ workarounds here. But had to see whether there is anything from the platform to help me out.
I am not sure LUKS would work, because we are talking about the file system in the Orderer container - any pointers would be good.
Restricting the access to the Orderer - Yes - That would work.
Ensuring the encryption of the data passed along - that also would work. And that is what I am going to do
jyellick (Tue, 06 Feb 2018 05:37:56 GMT):
@Ashish If the orderer process were to encrypt the ledger, then it would need to store the private key to decrypt it. Outside of specialized hardware, this means writing the key to the file system, so, if the attacker can read the files, then it does not really offer you any additional protections.
jyellick (Tue, 06 Feb 2018 05:38:17 GMT):
Maybe I am missing something, or what you are attempting to do?
Ashish (Tue, 06 Feb 2018 05:40:25 GMT):
On one of our reviews of data privacy, one associate pointed out that even if we are claiming the data is private in the ledger and with the help of channel permissioned access is implemented, if the intruder manages to get hold of the orderer he can practically see the invoker arguements.
Ashish (Tue, 06 Feb 2018 05:40:25 GMT):
@jyellick On one of our reviews of data privacy, one associate pointed out that even if we are claiming the data is private in the ledger and with the help of channel permissioned access is implemented, if the intruder manages to get hold of the orderer he can practically see the invoker arguements.
jyellick (Tue, 06 Feb 2018 05:43:27 GMT):
If the attacker compromises the orderer system to the extent that they can read arbitrary files, then I suspect the battle is already lost.
Ashish (Tue, 06 Feb 2018 05:44:21 GMT):
@jyellick Yes. I will give you that :)
Ashish (Tue, 06 Feb 2018 05:47:23 GMT):
the problem is not about the attacker from outside, its about the operatives inside who gets to see the data which they should not. And in Europe for financial systems, they are ensuring GDPR etc. So applications gets scanned thoroughly.
jyellick (Tue, 06 Feb 2018 05:50:53 GMT):
The options that spring to mind would be to either:
1. Use specialized hardware which restricts even administrators from performing operations like reads to the filesystem.
2. Design the data model not to reveal the regulated details (via application level encryption, ZK proofs, or other techniques)
3. Use something like "private data" so that the transaction affects are communicated peer to peer and do not pass through ordering
jyellick (Tue, 06 Feb 2018 05:50:53 GMT):
The options that spring to mind would be to either:
1. Use specialized hardware which restricts even administrators from performing operations like reads to the filesystem.
2. Design the data model not to reveal the regulated details (via application level encryption, ZK proofs, or other techniques)
3. Use something like "private data" so that the transaction effects are communicated peer to peer and do not pass through ordering
Ashish (Tue, 06 Feb 2018 05:56:59 GMT):
@jyellick Thank you very much,.
Point 2 This is most appropriate. I better guard my data at application layer itself. And this is what I intend to do readily.
Point 3 - with the Private data , i am guessing you are pointing to Side DB ? It had skipped my mind totally. But still it would be a case to case scenario.
Point 1 - we really have to think a bit into this. :(
manxiaqu (Thu, 08 Feb 2018 02:40:41 GMT):
Has joined the channel.
kostas (Thu, 08 Feb 2018 20:10:46 GMT):
Folks, please take a crack at the ordering service FAQ when you get a moment. I'd like to push it out by Monday.
guoger (Mon, 12 Feb 2018 16:51:17 GMT):
for orderer faq, on top of my head, we could add:
1. how to modify the configuration of orderer (consult configtxlator doc)
2. I encountered `Rejecting deliver request for %s because of consenter error` while creating channel, where should I start debugging? (AFAIK, we came across a lot of questions in this form, and most of them are caused by kafka connection problem...)
3. how should we plan OSN production deployment? network topology, organization, etc
guoger (Mon, 12 Feb 2018 16:51:45 GMT):
let me know if any of these should be added, I'll update the patch
guoger (Mon, 12 Feb 2018 16:51:56 GMT):
I'll keep thinking of new ones
kostas (Mon, 12 Feb 2018 16:55:14 GMT):
@guoger: All of these (maybe with the exception of item number 3, because I'm generally skeptical about going into production recommendations) would make good additions I think.
kostas (Mon, 12 Feb 2018 16:55:16 GMT):
Feel free to stub these QAs out via a patchset, and we can all modify them further if need be.
guoger (Mon, 12 Feb 2018 17:06:58 GMT):
Sure, will do my tomorrow. Btw, do we have this kinda ongoing FAQ work for every components?
kostas (Mon, 12 Feb 2018 17:10:59 GMT):
Not that I'm aware of.
kostas (Thu, 15 Feb 2018 13:46:49 GMT):
For the FAQ -- does my memory serve me well, in that there are incompatibilities between `configtxgen` and the artifacts it produces across versions? i.e. you cannot inspect an artifact produced by 1.0.0 with a 1.0.x binary, or something like that? If that is the case, @jyellick, can/should we identify those incompatibilities in a QA for the FAQ? (My cue for this is the latest message on the mailing list.)
jyellick (Thu, 15 Feb 2018 14:53:43 GMT):
@kostas There was an issue whee a v1.0.0 `configtxgen` did not produce channel creation txes which were valid for a v1.0.1+ orderer
jyellick (Thu, 15 Feb 2018 14:54:38 GMT):
And, because of the addition of new proto messages and config fields, the older one will not be able to decode output from the newer, though that seems somewhat intuitive to me
kostas (Thu, 15 Feb 2018 14:58:52 GMT):
Understood. Is the answer to the following one obvious -- can the newer binary encode/decode older artifacts just fine?
kostas (Thu, 15 Feb 2018 14:58:52 GMT):
Understood. Is the answer to the following one obvious -- can the newer binary decode older artifacts just fine?
jyellick (Thu, 15 Feb 2018 15:32:28 GMT):
Yes
jyellick (Thu, 15 Feb 2018 15:33:21 GMT):
(In fact, the newer one may even do a better job decoding older artifacts, as it knows of more opaque field types)
Pranoti (Fri, 16 Feb 2018 11:55:23 GMT):
Has joined the channel.
Pranoti (Fri, 16 Feb 2018 11:59:32 GMT):
Hi all,
I was executing this command
docker exec peer0.org1.example.com peer channel create -o orderer.example.com:7050 -c channel001 -f /etc/hyperledger/configtx/composer-channel.tx
Its throwing error. So I checked orderer logs I found this
WARN 0d6 Rejecting CONFIG_UPDATE because: Proposed configuration has no application group members, but consortium contains members
2018-02-16 10:21:22.969 UTC [orderer/main] func1 -> DEBU 0d7 Closing Broadcast stream
Does anybody have any idea about this error?
kostas (Fri, 16 Feb 2018 14:02:50 GMT):
@Pranoti: This channel is for dev discussions, and you're double-posting in #fabric-orderer. No bueno.
kostas (Fri, 16 Feb 2018 15:03:45 GMT):
Reposting from #fabric-scrum for reference: https://chat.hyperledger.org/channel/fabric-ci?msg=STyHFaHCDik7hj6zr
Cyrold (Fri, 16 Feb 2018 15:13:28 GMT):
Has joined the channel.
kostas (Fri, 16 Feb 2018 15:44:52 GMT):
Do we agree that this comment could use some love? https://github.com/hyperledger/fabric/blob/master/sampleconfig/configtx.yaml#L241..L242
jyellick (Fri, 16 Feb 2018 15:45:49 GMT):
Certainly
kostas (Fri, 16 Feb 2018 15:47:42 GMT):
I'm thinking "The name by which the org will be referenced in configuration transactions."
kostas (Fri, 16 Feb 2018 15:47:48 GMT):
But I'm not 100% sold on it.
kostas (Fri, 16 Feb 2018 15:47:57 GMT):
And I'm not sure if it's missing something.
kostas (Fri, 16 Feb 2018 15:48:10 GMT):
Once we settle on it I can push a quick CR addressing it.
kostas (Fri, 16 Feb 2018 15:49:57 GMT):
(I also question the need to have both a `Name` and `ID` field but that ship has sailed now.)
jyellick (Fri, 16 Feb 2018 15:51:31 GMT):
The rest of the comments follow the godoc style of repeating the variable name as the start of the comment
jyellick (Fri, 16 Feb 2018 15:52:14 GMT):
So, "Name is the key used to refer to the organization in channel configuration transactions." would be my first thought
Pranoti (Tue, 20 Feb 2018 13:57:04 GMT):
Has left the channel.
jyellick (Tue, 20 Feb 2018 20:06:27 GMT):
With respect to upgrade, we talk about upgrading the Kafka cluster, but not the ZK ensemble
jyellick (Tue, 20 Feb 2018 20:06:58 GMT):
The best doc I can seem to find for ZK is: https://wiki.apache.org/hadoop/ZooKeeper/FAQ#A6 and then a note for the beta only on: https://zookeeper.apache.org/doc/r3.5.3-beta/zookeeperReconfig.html#ch_reconfig_upgrade
jyellick (Tue, 20 Feb 2018 20:07:32 GMT):
Do we want to recommend that a ZK upgrade happens? Do we have any better doc to link them to?
jyellick (Tue, 20 Feb 2018 20:07:42 GMT):
@kostas @sanchezl @guoger ^
kostas (Tue, 20 Feb 2018 20:35:12 GMT):
^^ This one's uncharted territory for me. I defer to Luis.
sanchezl (Wed, 21 Feb 2018 14:29:57 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=4Yrp75EPKHvN6mL4c) @jyellick We point them to the kafka upgrade instructions, I defer the zookeeper upgrade to Apache Kafka's upgrade instructions.
sanchezl (Wed, 21 Feb 2018 14:31:31 GMT):
Basically, I don't think the ZK version matters much. I would expect most people to use the version bundled with their OS.
jyellick (Wed, 21 Feb 2018 14:32:56 GMT):
> , I defer the zookeeper upgrade to Apache Kafka's upgrade instructions.
The Kafka upgrade instructions do not even mention ZK
> Basically, I don't think the ZK version matters much. I would expect most people to use the version bundled with their OS.
My suspicion was that yes, barring a security fix we would not actually suggest users upgrade their ZK. However, I wasn't able to find anyone documenting this anywherre, do you have a source?
jyellick (Wed, 21 Feb 2018 14:36:26 GMT):
Apparently @yacovm knows some ZK experts who have managed it in mission critical scenarios. I asked him to pass along a set of questions and got back these responses:
> 1. Is keeping ZK up to date recommended at all? if the ensemble is functioning correctly (and there are no security bugs) should we just leave it alone?
> 2. Should ZK always be upgraded to the adjacent version, or may versions be skipped?
> 3. Should a rolling ZK upgrade be done in a maintenance window? Aside from the lost fault tolerance, can we expect any other degradation?
> 4. Is there any danger running a mixed ZK version ensemble? How quickly should the update roll?
> Of course the context for this is ZK for Kafka, where I would not expect ZK to be in the performance critical path
1. If it ain't broke... Keep the latest fully stable version that has the features and bug fixes you need. The latest is not necessarily the greatest when it comes to something like ZK. No alpha, no beta, wait until folks in the wild had time to find issues.
2. We have never skipped versions. Need to consult the documentation for each release, consult the committers to make sure there are no problems (e.g., clients changed in some way, etc).
3. Even a rolling upgrade provides opportunities for things to go wrong. Especially at the time when you take down the leader, things can go wrong. Various conditions can conspire against you to cause lengthy leader elections. It is rare, but we have seen it happen on more than one occasion. Even going as far as expired sessions. Especially if you are running on a less than stellar network infrastructure.
4. Never done that. I guess going from one version to the next one and so on should be fine. People cannot usually tolerate downtime. But again, consult docs and committers.
kostas (Thu, 22 Feb 2018 16:35:52 GMT):
My attempt to be cute with CI has backfired: https://gerrit.hyperledger.org/r/c/18129
kostas (Thu, 22 Feb 2018 16:36:08 GMT):
I was hoping that as long as a `reverify-x` was there in the comment, it would pick it up.
kostas (Thu, 22 Feb 2018 16:36:37 GMT):
It seems like it needs just the command on a comment of its own.
kostas (Thu, 22 Feb 2018 16:36:37 GMT):
~It seems like it needs just the command on a comment of its own.~ Nvm, it worked.
jyellick (Thu, 22 Feb 2018 17:11:22 GMT):
Unlike on push, the reverifies only seem to be picked up when there is room on the queue
jyellick (Thu, 22 Feb 2018 17:11:22 GMT):
Unlike on push, the reverifies only seem to be delayed until there is room on the queue
jyellick (Thu, 22 Feb 2018 17:11:22 GMT):
Unlike on push, the reverifies seem to be delayed until there is room on the queue'
jyellick (Thu, 22 Feb 2018 17:11:22 GMT):
Unlike on push, the reverifies seem to be delayed until there is room on the queue
kostas (Thu, 22 Feb 2018 18:29:41 GMT):
Can you remind me what's the way to do inline code in a JIRA comment? I'm drawing a blank right now.
jyellick (Thu, 22 Feb 2018 18:32:24 GMT):
`{{code}}`
jyellick (Thu, 22 Feb 2018 18:32:24 GMT):
@kostas `{{code}}`
kostas (Fri, 23 Feb 2018 18:15:25 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=3Ltenn3moTLFCpPov
kostas (Fri, 23 Feb 2018 18:18:38 GMT):
@guoger: Out of curiosity, did you figure this out? I was doing some `govendor` updates today locally and I also see what Luis wrote here: https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=c2XMKkFPuLmGPYws7
jyellick (Fri, 23 Feb 2018 18:32:56 GMT):
@sanchezl What is the status of https://jira.hyperledger.org/browse/FAB-8408 ?
sanchezl (Fri, 23 Feb 2018 20:24:10 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=DRxhe22kguRR62AqR) @jyellick I'll make a comment. It's not our problem.
SashiKanth (Sat, 24 Feb 2018 08:05:59 GMT):
Has joined the channel.
SashiKanth (Sat, 24 Feb 2018 08:06:03 GMT):
Hi,
I got this error when connecting a peer to the already exsiting fabric network, but the peer resides in different host
Failed to dial 10.0.0.6:7050: connection error: desc = "transport: authentication handshake failed: x509: cannot validate certificate for 10.0.0.6 because it doesn't contain any IP SANs"; please retry.
kostas (Sat, 24 Feb 2018 14:11:49 GMT):
SashiKanth
kostas (Sat, 24 Feb 2018 14:11:55 GMT):
User User_2 removed by kostas.
debutinfotech (Wed, 28 Feb 2018 09:26:33 GMT):
Has joined the channel.
debutinfotech (Wed, 28 Feb 2018 09:27:35 GMT):
what is the difference between solo and orderer?
debutinfotech (Wed, 28 Feb 2018 09:27:35 GMT):
what is the difference between solo and Kafka?
chandrakanthm (Wed, 28 Feb 2018 09:55:05 GMT):
Has joined the channel.
pankajcheema (Wed, 28 Feb 2018 11:17:02 GMT):
Has joined the channel.
kostas (Wed, 28 Feb 2018 12:52:14 GMT):
debutinfotech
kostas (Wed, 28 Feb 2018 12:52:21 GMT):
User User_3 removed by kostas.
pankajcheema (Mon, 05 Mar 2018 11:47:45 GMT):
s
kostas (Tue, 06 Mar 2018 15:59:29 GMT):
What are our thoughts on modifying `configtxgen` so that it accepts `-config` parameter?
kostas (Tue, 06 Mar 2018 16:00:19 GMT):
When set it would override the existing configuration mechanism (which looks for a file in `FABRIC_CFG_PATH` if set, then CWD, etc.)
kostas (Tue, 06 Mar 2018 16:01:32 GMT):
This would also make the `configtxgen` behavior resemble that one of `cryptogen`.
kostas (Tue, 06 Mar 2018 16:01:32 GMT):
This would also make the `configtxgen` behavior resemble that of `cryptogen`.
jyellick (Tue, 06 Mar 2018 16:54:47 GMT):
Definitely in favor, I thought I might have even opened an issue for this
jyellick (Tue, 06 Mar 2018 16:57:08 GMT):
Looks like I did not, but yes, I think this is an easy should-add for v1.2
dave.enyeart (Tue, 06 Mar 2018 17:06:48 GMT):
Has joined the channel.
rjones (Tue, 06 Mar 2018 18:15:19 GMT):
dave.enyeart
guoger (Wed, 07 Mar 2018 03:50:37 GMT):
looking at https://jira.hyperledger.org/browse/FAB-8649, I contacted reporter and told him the correct configuration. However, `ORDERER_KAFKA_TLS` and `ORDERER_GENERAL_TLS` essentially share the same data structure, whereas the former expects pem key (or file path under extra `File` key), and the latter expects file path. Even though we document this in `orderer.yaml`, I feel it's still not intuitive and may cause confusion. Furthermore, we don't allow multiple `rootCA` in `ORDERER_KAFKA_TLS`, but does that for `ORDERER_GENERAL_TLS`. I think we probably should unify the parsing logic for both sections. Thoughts? @sanchezl @kostas
kostas (Wed, 07 Mar 2018 04:19:46 GMT):
@guoger: I think you are right and I'd vote for unifying that logic.
kostas (Wed, 07 Mar 2018 04:20:56 GMT):
> Looks like I did not, but yes, I think this is an easy should-add for v1.2
I was writing up another JIRA, and I opened up this one as well so that it doesn't slip through the cracks: https://jira.hyperledger.org/browse/FAB-8686
kostas (Wed, 07 Mar 2018 04:21:22 GMT):
That other JIRA by the way is: https://jira.hyperledger.org/browse/FAB-8685 - I'd love to hear your thoughts on this.
guoger (Wed, 07 Mar 2018 05:18:41 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=6BR7S3h6FEdm5sMAg) sounds great, should I tell reporter to reopen the issue and change it to improvement and target 1.1.1 maybe?
guoger (Wed, 07 Mar 2018 05:20:34 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=vf3QYicaPGPH7tzkC) @kostas +1 for this. just wondering if we benchmark consensus algorithm and wanna make sure I/O is not the bottleneck, is `ram ledger` kinda useful in this case?
kostas (Wed, 07 Mar 2018 14:42:26 GMT):
> @kostas +1 for this. just wondering if we benchmark consensus algorithm and wanna make sure I/O is not the bottleneck, is `ram ledger` kinda useful in this case?
Perhaps. Another benefit of the RAM ledger is that currently we can do `go build && ./orderer` and have an orderer instance up and running for tests without having to worry about leftover artifacts. But I would claim that these are all small benefits and not worth keeping these additional "for-testing" implementations in our master tree.
kostas (Wed, 07 Mar 2018 14:42:26 GMT):
> @kostas +1 for this. just wondering if we benchmark consensus algorithm and wanna make sure I/O is not the bottleneck, is `ram ledger` kinda useful in this case?
Perhaps. Another benefit of the RAM ledger is that currently we can do `go build && ./orderer` and have an orderer instance up and running for tests without having to worry about leftover artifacts. But I would claim that these are all small benefits and it's not worth keeping these additional "for-testing" implementations in our master tree any more.
kostas (Wed, 07 Mar 2018 14:47:38 GMT):
> sounds great, should I tell reporter to reopen the issue and change it to improvement and target 1.1.1 maybe?
I think that's a good idea. FWIW, I wouldn't set the "fix" target to 1.1.1 unless a contributor has cycles to take this over and is ready to assign it to themselves. I think we should default to "Backlog" for all new items from now on.
jyellick (Wed, 07 Mar 2018 15:01:08 GMT):
Debated just posting it to the issue, but as it is 'help-wanted' I did not want to confuse the issue, but with respect to FAB-8685:
I can certainly see removing the option from `orderer.yaml` but are we sure there's no value in leaving at least the RAM ledger implementation around? In particular, for doing things like integration tests with multiple logical orderer processes, it still seems like a handy thing to keep around. Are we actively encountering problems maintaining them? Are we trying to change the ledger interface or similar?
kostas (Wed, 07 Mar 2018 15:18:39 GMT):
@jyellick: I have thoughts on this. Let's have the debate in the JIRA? It'll be useful to record our thoughts somewhere more permanent.
jyellick (Wed, 07 Mar 2018 15:19:23 GMT):
Sure
kostas (Wed, 07 Mar 2018 15:38:01 GMT):
Responded there.
kostas (Wed, 07 Mar 2018 15:38:48 GMT):
I've taken a stab at cleaning the backlog a bit. If I've categorized an item you own inappropriately, please feel free to edit back.
pankajcheema (Mon, 12 Mar 2018 04:48:40 GMT):
Anyone here know how to find `CORE_PEER_LOCALMSPID`?
jyellick (Mon, 12 Mar 2018 04:50:31 GMT):
pankajcheema
jyellick (Mon, 12 Mar 2018 04:50:39 GMT):
User User_4 removed by jyellick.
MoulaliMvg (Mon, 12 Mar 2018 05:22:35 GMT):
Has joined the channel.
guoger (Mon, 12 Mar 2018 09:49:03 GMT):
Rumor has it that we plan to transform some of system chaincode to be scc plugins (IIUC, https://jira.hyperledger.org/browse/FAB-6719 is repealed?). Do you know who's working on this piece of work? I encountered a problem while working with evm integration (which is implemented as a scc plugin), more specifically, it's documented in go-nuts mailing list here: https://groups.google.com/forum/#!topic/golang-nuts/66VupPd_2ZU
guoger (Mon, 12 Mar 2018 09:50:00 GMT):
I suspect there will be similar problems within other scc plugins and I'm seeking helps/suggestions/collaborations to solve this :)
jyellick (Mon, 12 Mar 2018 13:36:10 GMT):
@guoger This is going to be @yacovm primarily I believe
yacovm (Mon, 12 Mar 2018 13:53:56 GMT):
hold on
yacovm (Mon, 12 Mar 2018 13:54:02 GMT):
I'm doing work on ESCC and VSCC
yacovm (Mon, 12 Mar 2018 13:54:07 GMT):
it has nothing to do with EVM
yacovm (Mon, 12 Mar 2018 13:54:30 GMT):
also it doesn't mean that FAB-6719 is repealed
yacovm (Mon, 12 Mar 2018 14:10:03 GMT):
@troyronda you have experience with writing SCCs as plugins
troyronda (Mon, 12 Mar 2018 14:10:03 GMT):
Has joined the channel.
yacovm (Mon, 12 Mar 2018 14:10:10 GMT):
any thoughts about the above?
guoger (Mon, 12 Mar 2018 14:12:51 GMT):
Just a bit more context, it's very likely that scc plugin and Fabric both vendor `grpc`, which vendors `golang.org/x/net/trace`, which registers to `/debug/requests` endpoint, and we then have double-registration, and it panics
troyronda (Mon, 12 Mar 2018 16:47:53 GMT):
@yacovm @guoger (cc: @divyank) yup we have successfully built scc plugins for fabric
divyank (Mon, 12 Mar 2018 16:47:53 GMT):
Has joined the channel.
troyronda (Mon, 12 Mar 2018 16:48:14 GMT):
we have a script process to deal with vendoring within the plugins
troyronda (Mon, 12 Mar 2018 16:48:14 GMT):
we have a script to deal with vendoring within the plugins to work with the specific fabric version we are targetting
troyronda (Mon, 12 Mar 2018 16:48:14 GMT):
we have a script to deal with vendoring within the plugins to work with the fabric version we are targeting
troyronda (Mon, 12 Mar 2018 16:48:14 GMT):
we have a script to deal with vendoring within the plugins to work with the fabric version we are targeting, and to build the plugin using the go version that fabric was built with
troyronda (Mon, 12 Mar 2018 16:48:14 GMT):
we have a script to deal with vendoring within the plugins to work with the fabric version we are targeting (and to build the plugins)
troyronda (Mon, 12 Mar 2018 16:54:32 GMT):
we also had to apply a small patch to the Go 1.9.2 compiler (can go away once fabric is on Go 1.10)
troyronda (Mon, 12 Mar 2018 16:54:32 GMT):
we also had to apply a small patch to the Go 1.9.2 compiler that builds the plugins (this patch can go away once fabric is on Go 1.10)
troyronda (Mon, 12 Mar 2018 17:00:43 GMT):
also: for some special cases, where a plugin (A) is referencing code loaded in another plugin (B), a small patch is need for the Go 1.9 compiler that builds the (A) plugin. This patch is not needed when Fabric is build with Go 1.10.
troyronda (Mon, 12 Mar 2018 17:00:43 GMT):
also: for some special cases, where a plugin (A) is referencing code loaded in another plugin (B), a small patch is need for the Go 1.9 compiler that builds the (A) plugin. This patch is not needed when Fabric is built with Go 1.10.
troyronda (Mon, 12 Mar 2018 17:00:43 GMT):
also: for some special cases, where a plugin (A) is referencing code loaded in another plugin (B), a small patch is need for the Go 1.9.2 compiler that builds the (A) plugin. This patch is not needed when Fabric is built with Go 1.10.
troyronda (Mon, 12 Mar 2018 17:00:43 GMT):
also: for some special cases, where a plugin (A) is referencing code loaded in another plugin (B), a small patch is need for the Go 1.9.2 compiler that builds the (A) plugin. This patch is not needed when Fabric and plugins are built with Go 1.10.
troyronda (Mon, 12 Mar 2018 17:00:43 GMT):
also: for some special cases, where a plugin (A) is referencing code loaded in another plugin (B), a small patch is need for the Go 1.9.2 compiler that builds the (A) plugin. This patch is not needed when Fabric and plugins are built with Go 1.10 (but we haven't tested on 1.10 yet).
troyronda (Mon, 12 Mar 2018 17:00:43 GMT):
also: for some special cases, where a plugin (A) is referencing code loaded in another plugin (B), a small patch is needed for the Go 1.9.2 compiler that builds the (A) plugin. This patch is not needed when Fabric and plugins are built with Go 1.10 (but we haven't tested on 1.10 yet).
yacovm (Mon, 12 Mar 2018 17:04:37 GMT):
Oh thats great so on theory once we move to 1.10 we dont have double loading?
yacovm (Mon, 12 Mar 2018 17:04:44 GMT):
"in
troyronda (Mon, 12 Mar 2018 17:11:00 GMT):
Unfortunately I hear that you still need to deal with flattening the vendored dependencies if that's what you are referring to
troyronda (Mon, 12 Mar 2018 17:11:00 GMT):
Unfortunately I hear that you still need to deal with flattening the vendored dependencies (and ensuring the right version) if that's what you are referring to
troyronda (Mon, 12 Mar 2018 17:11:00 GMT):
Unfortunately I hear that you still need to deal with ensuring the right (matching) version of the vendored dependencies if that's what you are referring to
troyronda (Mon, 12 Mar 2018 17:11:36 GMT):
(I haven't tried it yet)
troyronda (Mon, 12 Mar 2018 17:14:27 GMT):
@divyank knows a bit more about it
divyank (Mon, 12 Mar 2018 17:18:41 GMT):
The issue that's fixed with 1.10 prevents a plugin from importing code that calls plugin.Open() (like the BCCSP package) https://github.com/golang/go/issues/22175
(We used the compiler patch to work around that in 1.9.2)
divyank (Mon, 12 Mar 2018 17:20:24 GMT):
Unfortunately there are no fixes in progress for the double loading issues. We use scripts to flatten imported libraries on the path so that only one copy exists at runtime.
troyronda (Mon, 12 Mar 2018 17:20:39 GMT):
thanks divyank
guoger (Tue, 13 Mar 2018 01:29:39 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=3YqydYsqXEfm7BC6L) @divyank Could you elaborate on "flatten imported libraries"? or could you share the script that I could take a peek at? thx!
GopalPanda (Tue, 13 Mar 2018 01:54:22 GMT):
Has joined the channel.
swetha (Tue, 13 Mar 2018 02:26:04 GMT):
Has joined the channel.
divyank (Tue, 13 Mar 2018 17:25:38 GMT):
@guoger Sure, the Go runtime considers host/vendor/libA and plugin/vendor/libA as distinct packages. As you've noted above, certain packages like golang/x/net cannot co-exist with other instances of themselves. So by flatten I mean that we delete libraries in plugin/vendor that already exist in host/vendor: https://github.com/securekey/fabric-snaps/blob/master/scripts/flatten_deps.sh
Additionally, during plugin compilation, we embed plugins within fabric. This ensures that the plugin picks up libraries in host/vendor: https://github.com/securekey/fabric-snaps/blob/master/scripts/move_snaps.sh
ShikarSharma (Tue, 20 Mar 2018 22:43:57 GMT):
Has joined the channel.
guoger (Wed, 21 Mar 2018 07:24:44 GMT):
Just wonder, shouldn't this line https://github.com/hyperledger/fabric-baseimage/blob/master/Makefile#L10 be `IS_RELEASE ?= false`?
kostas (Fri, 23 Mar 2018 16:16:59 GMT):
@guoger: I have no idea (my Makefile/bash-fu is weak). Maybe ask in #fabric-ci?
troyronda (Fri, 23 Mar 2018 17:06:15 GMT):
@guoger they use that variable to denote releases to the CI (and hence, it wouldn't need to be overridable).
patelan (Fri, 23 Mar 2018 19:37:19 GMT):
Has joined the channel.
patelan (Fri, 23 Mar 2018 19:37:37 GMT):
Hi All, QQ. Do we have any keepalive setting between peer and orderer ? We are using fabric 1.0.3
jyellick (Fri, 23 Mar 2018 19:38:27 GMT):
@patelan This channel is for development discussions only, please ask general usage questions in #fabric-orderer
jyellick (Fri, 23 Mar 2018 19:38:48 GMT):
And do not cross post
patelan (Fri, 23 Mar 2018 19:38:57 GMT):
@jyellick okey sure Thanks
patelan (Fri, 23 Mar 2018 19:40:17 GMT):
Has left the channel.
wbhagan (Tue, 27 Mar 2018 16:06:58 GMT):
Has joined the channel.
JiuZhuYou (Sat, 31 Mar 2018 10:40:56 GMT):
Has joined the channel.
richzhao (Sun, 01 Apr 2018 15:19:59 GMT):
Has joined the channel.
Rumeel_Hussain (Tue, 03 Apr 2018 15:11:01 GMT):
Has joined the channel.
kostas (Thu, 05 Apr 2018 16:34:04 GMT):
This is clever: https://blog.antoine-augusti.fr/2018/03/golang-instant-first-tick-for-ticker/
jyellick (Thu, 05 Apr 2018 17:05:44 GMT):
Clever, but I think my preference would be:
```for {
fmt.Println("Tick at: ",time.Now())
if _, ok := <- ticker.C ; !ok {
break
}
}
```
jyellick (Thu, 05 Apr 2018 17:05:44 GMT):
Clever, but I think my preference would be:
```for {
fmt.Println("Tick at: ",time.Now())
if _, ok := <- ticker.C ; !ok {
break
}
}
```
(purely because it is not clever)
kostas (Thu, 05 Apr 2018 18:47:15 GMT):
^^ That is quite nice as well.
jyellick (Thu, 05 Apr 2018 19:10:29 GMT):
And actually, to accomplish the same effect as the link, you could simply do:
```for {
fmt.Println("Tick at: ", time.Now())
<-ticker.C
}
```
(The for-loop version does not actually exit when the ticker is stopped)
jyellick (Thu, 05 Apr 2018 19:10:29 GMT):
And actually, to accomplish the same effect as the link, you could simply do:
```for {
fmt.Println("Tick at: ", time.Now())
<-ticker.C
}
```
(The for-loop version does not actually exit when the ticker is stopped -- it would in fact spin continuously I believe)
kostas (Thu, 05 Apr 2018 19:28:37 GMT):
You are right, it would spin forever.
kostas (Mon, 16 Apr 2018 14:20:21 GMT):
FYI: https://jira.hyperledger.org/browse/FAB-9494
kostas (Mon, 23 Apr 2018 21:37:52 GMT):
I'm not sure if it goes against common best practices, but my mildly hot take for the day is that our attempts to translate relative paths into absolute paths in our YAML files is a bad idea.
kostas (Mon, 23 Apr 2018 21:38:51 GMT):
Just have the user go with absolute paths and call it a day.
jyellick (Tue, 24 Apr 2018 01:42:46 GMT):
Interesting, how would you handle two different dev environments in different go paths?
kostas (Tue, 24 Apr 2018 14:13:33 GMT):
Not sure I follow?
jyellick (Tue, 24 Apr 2018 14:14:52 GMT):
So for instance, in our `orderer.yaml` we specify an MSP path. If my gopath is `/home/jyellick/go/` and your go path is `/Users/kostas/go` how do we make the `sampeconfig/orderer.yaml` portable? Are you suggesting we don't?
kostas (Tue, 24 Apr 2018 14:26:25 GMT):
Correct. I'd adjust the file path references so that they match my filesystem, and you'd do the same for yours. (I did say it's a mildly hot take.)
kostas (Wed, 25 Apr 2018 15:12:49 GMT):
#fabric-orderer, and heck RC in its entirety, needs to be shut down. It is the path of least resistance so _of course_ this is where users will post questions, but it incentivizes bad behavior. Folks just don't take the time to formulate their questions appropriately. That 2-3 line text-field is really mean for quick one-offs, and user questions are _rarely_ that. So you end up with a casually and hastily written description of a rather complicated problem. And given the inability to threading, you have these seemingly out-of-context "@jyellick any solution to my problem?" messages, which are actually references to a message posted 6 days ago. A truly awful venue for user questions.
kostas (Wed, 25 Apr 2018 15:12:49 GMT):
#fabric-orderer, and heck RC in its entirety, needs to be shut down. It is the path of least resistance so _of course_ this is where users will post questions, but it incentivizes bad behavior. Folks just don't take the time to formulate their questions appropriately. That 2-3 line text-field is really meant for quick one-liners, and user questions are _rarely_ that. So you end up with a casually and hastily written description of a rather complicated problem. And given the inability to threading, you have these seemingly out-of-context "@jyellick any solution to my problem?" messages, which are actually references to a message posted 6 days ago. A truly awful venue for user questions.
kostas (Wed, 25 Apr 2018 15:13:46 GMT):
I am ranting now, but my plan is to actually propose this to the mailing list eventually. (Folks will shoot it down, I know.)
kostas (Wed, 25 Apr 2018 15:14:21 GMT):
Chat only makes sense for quick syncing between devs. Keep all the dev channels, kill anything else.
kostas (Wed, 25 Apr 2018 15:14:21 GMT):
RC really only makes sense for quick syncing between devs. Keep all the dev channels, kill anything else.
jyellick (Wed, 25 Apr 2018 15:26:57 GMT):
Answer on SO is certainly a more useful exercise
kostas (Wed, 25 Apr 2018 15:43:59 GMT):
My point is, as long as RC exists as a venue for user questions, expect SO (or the mailing list) to be used as a last resort.
kostas (Wed, 25 Apr 2018 15:43:59 GMT):
As long as RC exists as a venue for user questions, we'll be seeing SO (or the mailing list) used as a last resort.
kostas (Wed, 25 Apr 2018 15:44:33 GMT):
It's not the users' fault. It's ours for offering this option.
chainsaw (Fri, 27 Apr 2018 15:53:08 GMT):
Has joined the channel.
kostas (Tue, 01 May 2018 17:05:19 GMT):
Had accidentally bumped into those before and was always too rushed to check them out. GitHub's keyboard shortcuts are good timesavers: press `?` whenever you're in a repo.
kostas (Tue, 01 May 2018 17:05:19 GMT):
Had accidentally bumped into those before and was always too rushed to check them out. GitHub's keyboard shortcuts are good timesavers: press `?` whenever you're browsing a repo.
jyellick (Tue, 01 May 2018 17:13:36 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=Xo59KAWHkZs92atXy
jyellick (Tue, 01 May 2018 17:13:50 GMT):
Related: I installed this when you had posted the above, continue to find it very useful.
kostas (Tue, 01 May 2018 17:21:09 GMT):
Ooh nice, good to know.
kostas (Tue, 01 May 2018 17:21:42 GMT):
I want to use their search engine more but my lack of regex-fu isn't helping: https://sourcegraph.com/search
kevin-s-wang (Thu, 03 May 2018 02:37:05 GMT):
Has joined the channel.
binhn (Thu, 03 May 2018 13:13:17 GMT):
Has left the channel.
kostas (Fri, 04 May 2018 15:17:21 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=KgMSGuAubkgXWf9ia
kostas (Fri, 04 May 2018 15:17:54 GMT):
I want this pinned to this room, posted as a sign right outside my house, and written on my tombstone.
jyellick (Fri, 04 May 2018 18:24:46 GMT):
FYI: https://status.linuxfoundation.org/incidents/zjkddd9wt0yt
Lexliw (Sat, 05 May 2018 17:37:11 GMT):
Has joined the channel.
versus (Mon, 14 May 2018 09:06:15 GMT):
Has joined the channel.
dave.enyeart (Wed, 16 May 2018 11:55:44 GMT):
@jyellick If I enable orderer capability v1_2:true, I get the following error:
dave.enyeart (Wed, 16 May 2018 11:55:50 GMT):
```2018-05-16 07:53:06.911 EDT [orderer/commmon/multichannel] checkResourcesOrPanic -> CRIT 3a8 [channel test-system-channel-name] config requires unsupported orderer capabilities: Orderer capability V1_2 is required but not supported: Orderer capability V1_2 is required but not supported```
dave.enyeart (Wed, 16 May 2018 11:56:10 GMT):
latest master
dave.enyeart (Wed, 16 May 2018 11:56:24 GMT):
is it not yet ready for v1_2 capability? or did i do something wrong?
dave.enyeart (Wed, 16 May 2018 11:57:45 GMT):
starting orderer with:
dave.enyeart (Wed, 16 May 2018 11:57:48 GMT):
`ORDERER_GENERAL_GENESISPROFILE=SampleSingleMSPSolo orderer`
jyellick (Wed, 16 May 2018 12:56:43 GMT):
@dave.enyeart There is no orderer v1_2 capability
jyellick (Wed, 16 May 2018 12:57:07 GMT):
There is only an application v1_2 capability. There are no non-backwards compatible changes to the orderer which require a capability this release.
dave.enyeart (Wed, 16 May 2018 13:03:57 GMT):
@jyellick thanks, that explains it! Can you make sure that is clear in the upgrade doc? (i didnt check, may be something there already...)
jyellick (Wed, 16 May 2018 13:34:04 GMT):
Will do
kostas (Tue, 22 May 2018 18:35:20 GMT):
Latest master failing consistently?
kostas (Tue, 22 May 2018 18:35:22 GMT):
```2018-05-22 18:33:33.150 UTC [chaincodeCmd] checkChaincodeCmdParams -> INFO 010 Using vscc vscc
unit-tests_1 | FAIL github.com/hyperledger/fabric/peer/chaincode 35.657s```
DivyaAgrawal (Tue, 22 May 2018 19:51:09 GMT):
Has joined the channel.
yacovm (Wed, 23 May 2018 19:17:46 GMT):
@kostas - ever since @sykesm fixed all kinds of fabric bugs, i just run UTs via `go test ./...` and that's it.
sykesm (Wed, 23 May 2018 19:17:46 GMT):
Has joined the channel.
yacovm (Wed, 23 May 2018 19:18:32 GMT):
much faster, no need to wait for the docker container to be built, and if there are failures that are irrelevant i just ignore them.
sykesm (Wed, 23 May 2018 19:20:58 GMT):
https://gerrit.hyperledger.org/r/c/22211/ / FAB-10334
nvmadhav (Fri, 25 May 2018 02:38:41 GMT):
Has joined the channel.
kostas (Fri, 25 May 2018 16:18:00 GMT):
That's a strange way to use log statements: https://github.com/docker/swarmkit/blob/master/manager/state/raft/raft.go#L935
kostas (Fri, 25 May 2018 16:18:28 GMT):
I can see what purpose it serves, but I don't think I've ever seen that before. Interesting.
jyellick (Fri, 25 May 2018 16:18:50 GMT):
What's the purpose, just as a visual start/stop queue?
kostas (Fri, 25 May 2018 16:19:14 GMT):
Yup, that's my guess. A way for the developer (given that it's a debug statement) to know that execution has hit that line.
guoger (Fri, 25 May 2018 16:29:40 GMT):
is it an empty line?
kostas (Fri, 25 May 2018 17:10:19 GMT):
Correct.
Aswath8687 (Mon, 28 May 2018 04:10:24 GMT):
Has joined the channel.
rogerwilcos (Wed, 30 May 2018 23:12:48 GMT):
Has joined the channel.
kostas (Thu, 31 May 2018 01:54:19 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=3icbD6pjg5MxWmxH7
kostas (Thu, 31 May 2018 01:54:29 GMT):
This one's particularly useful: https://twitter.com/derrickreimer/status/1001842631914254336
jyellick (Thu, 31 May 2018 01:55:11 GMT):
That is incredibly useful, I hate passing out links to master
jyellick (Thu, 31 May 2018 01:55:50 GMT):
I suppose it was too much to ask to yank it to the clipboard as well
kostas (Thu, 31 May 2018 01:55:58 GMT):
Heh, I'd say so.
guoger (Thu, 31 May 2018 06:02:09 GMT):
when is code freeze for v1.2? thx
jyellick (Thu, 31 May 2018 06:02:44 GMT):
@guoger I haven't heard about any official code freeze, but we are definitely locked down for new feature
jyellick (Thu, 31 May 2018 06:02:44 GMT):
@guoger I haven't heard about any official code freeze, but we are definitely locked down for new features
jyellick (Thu, 31 May 2018 06:03:28 GMT):
Ideally only tests, doc, and bug fixes should be going in at this point. There are possibly some trailing CRs required for some already accepted into release feature
guoger (Thu, 31 May 2018 06:03:45 GMT):
gotcha, thx!!
ashishchainworks (Fri, 01 Jun 2018 12:48:25 GMT):
Has joined the channel.
ashishchainworks (Fri, 01 Jun 2018 12:48:29 GMT):
Hi, This is regarding signature of Orderer in block metadata. In case of multiple OSNs in a network, whose signature will be there in block. If it is of OSN cutting the block, will that not make blocks in network different (though only by signature). Pls help.
jyellick (Mon, 04 Jun 2018 14:24:16 GMT):
https://gerrit.hyperledger.org/r/c/22725/
Bug in v1.1 which causes batch size changes only to occur on orderer restart
dappcoder (Tue, 05 Jun 2018 14:03:20 GMT):
Has joined the channel.
minollo (Tue, 05 Jun 2018 14:14:58 GMT):
Has joined the channel.
Ryan2 (Tue, 05 Jun 2018 22:37:04 GMT):
Has joined the channel.
abraham (Fri, 08 Jun 2018 05:39:00 GMT):
Has joined the channel.
knagware9 (Sat, 09 Jun 2018 10:20:07 GMT):
Has joined the channel.
MarcelvandeKerkhof (Tue, 12 Jun 2018 10:16:29 GMT):
Has joined the channel.
paulananth (Fri, 15 Jun 2018 12:19:16 GMT):
Has joined the channel.
kostas (Tue, 19 Jun 2018 23:47:56 GMT):
Bumped into this one today: https://github.com/golang/lint/issues/258
guoger (Wed, 20 Jun 2018 02:31:22 GMT):
hmmm... interesting...
kostas (Wed, 20 Jun 2018 02:36:34 GMT):
(Wasn't aware of the concept of Hungarian notation either which is a faux-pas I've definitely committed.)
kostas (Thu, 21 Jun 2018 01:31:25 GMT):
Is there a way to move an existing Story under an Epic?
guoger (Thu, 21 Jun 2018 01:34:07 GMT):
@kostas chris told me to help with raft :) so I wanna sync up with you to see where I should start
kostas (Thu, 21 Jun 2018 01:34:43 GMT):
Right, I asked him if you were available, and he told me the good news. I'm glad we'll have you working with us again :)
kostas (Thu, 21 Jun 2018 01:35:58 GMT):
Jay, we'll post this to the mailing list at the end of this week, or early next one, so let's not publicize it much until then (though I do get I'm posting this in a public channel). These are our design notes on the Raft integration: https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit
kostas (Thu, 21 Jun 2018 01:36:26 GMT):
Feel free to have a look when you get a chance, and leave comments, point out flaws, etc.
kostas (Thu, 21 Jun 2018 01:37:45 GMT):
The rough timeline is: post in 2-3d, collect/address feedback for next 2w, and in the meantime create small, easily digestible stories. Then we kick this thing off w/ implementation.
kostas (Thu, 21 Jun 2018 01:37:45 GMT):
The rough timeline is: post publicly in 2-3d, collect/address feedback for next 2w, and in the meantime create small, easily digestible stories. Then we kick this thing off w/ implementation.
kostas (Thu, 21 Jun 2018 01:37:45 GMT):
The rough timeline is: post publicly in 2-3d, collect/address feedback for next 2w, and in the meantime I'll do my best to create small, easily digestible stories. Then we kick this thing off w/ implementation.
kostas (Thu, 21 Jun 2018 01:37:45 GMT):
The rough timeline is: post publicly in 2-3d, collect/address feedback for next 2w, and in the meantime I'll do my best to create small, easily digestible stories. Then we kick this thing off w/ implementation. I will post again to the mailing list in 2-3w from now with links to the stories, asking interested folks to help out.
kostas (Thu, 21 Jun 2018 01:40:13 GMT):
@dappcoder: See above ^^
kostas (Thu, 21 Jun 2018 01:40:13 GMT):
@dappcoder: See above ^^ (I had told you I'd let you know once we had that Raft integration document out)
guoger (Thu, 21 Jun 2018 01:41:07 GMT):
fantastic! I'll look into the design doc
guoger (Thu, 21 Jun 2018 01:41:39 GMT):
I immediately thought about reusing etcd raft implementation when I was told about this task
kostas (Thu, 21 Jun 2018 01:41:44 GMT):
@gombiuda: You had reached out to me expressing interest in contributing to the Raft work. See my notes above.
gombiuda (Thu, 21 Jun 2018 01:41:44 GMT):
Has joined the channel.
guoger (Thu, 21 Jun 2018 01:41:54 GMT):
instead of writing from scratch
kostas (Thu, 21 Jun 2018 01:42:04 GMT):
@guoger: Perfect, glad we're on the same page.
jyellick (Thu, 21 Jun 2018 16:12:11 GMT):
https://github.com/pkg/errors/blob/master/errors.go#L198-L200
Thought I would share, as I found this behavior to be quite unintuitive and just spent quite a bit of time debugging trying to figure out why when I was clearly returning errors.Wrapf, the returned thing was nil.
Event (Sun, 24 Jun 2018 08:42:48 GMT):
Has joined the channel.
Event (Sun, 24 Jun 2018 08:42:54 GMT):
Hi, I am new to this chat and Hyperledger.
Are there any coders and UI Developers interested to collaborate on an exciting project?
I have entire business logic but need help with coding and making a functional UI, as first step. Any suggestions are most welcome. Thanks.
guoger (Mon, 25 Jun 2018 15:04:16 GMT):
@Event this channel is use for orderer development discussion *only*. you probably will get a better answer if you send your request to Fabric mailing list at fabric@lists.hyperledger.org
Event (Mon, 25 Jun 2018 16:29:44 GMT):
ok Thnx @guoger
DivyaAgrawal (Tue, 26 Jun 2018 09:06:09 GMT):
Hello all,
If i add some debug/logging statements in the orderer and build it
and copy the new binary to the bin location in fabric-sample
on running the first-network will new orderer be executed? or I need to something else too?
TIA
kostas (Tue, 26 Jun 2018 20:34:46 GMT):
So [in this article from 2011](https://arstechnica.com/science/2011/04/guns-in-the-home-lots-of-risk-ambiguity/), the author slipped in the following sentence:
> If you have read this far, please mention Bananas in your comment below. We're pretty sure 90% of the respondents to this story won't even read it first.
kostas (Tue, 26 Jun 2018 20:34:46 GMT):
So [in this article from 2011](https://arstechnica.com/science/2011/04/guns-in-the-home-lots-of-risk-ambiguity/), the author asked readers to mention the word 'bananas' in their comment, if they had read this far. It took 92 comments before someone did write 'bananas'. I wanted to try something similar with the Raft doc but I forgot.
kostas (Tue, 26 Jun 2018 20:34:56 GMT):
I wanted to do something similar for the Raft doc but I forgot.
kostas (Tue, 26 Jun 2018 20:35:30 GMT):
(It took 92 comments before someone wrote 'bananas' in that article BTW.)
kostas (Tue, 26 Jun 2018 20:35:30 GMT):
(It took 92 comments before someone wrote 'bananas' BTW.)
toddinpal (Tue, 26 Jun 2018 23:40:32 GMT):
Has joined the channel.
toddinpal (Tue, 26 Jun 2018 23:42:59 GMT):
Any thought about using a BFT hardened Raft such as Tangaroa instead of straight Raft?
kostas (Tue, 26 Jun 2018 23:54:28 GMT):
Is there a library for it with the same qualities as etcd/raft? (Go, right license, nicely documented, polished, well-tested.)
kostas (Tue, 26 Jun 2018 23:54:28 GMT):
Is there a library for it with the same qualities as etcd/raft? (Go, right license, nicely documented, polished, well-tested.) I do not seem to find any.
toddinpal (Wed, 27 Jun 2018 00:05:48 GMT):
I can look...
puneetsharma86 (Wed, 27 Jun 2018 09:38:16 GMT):
Has joined the channel.
kostas (Tue, 03 Jul 2018 01:39:14 GMT):
```
func main() {
a := []int{0, 1}
fmt.Printf("%v", a[len(a):])
}
```
kostas (Tue, 03 Jul 2018 01:39:32 GMT):
Does this panic (index out of range), or not? Taking bets.
jyellick (Tue, 03 Jul 2018 02:52:07 GMT):
My money was on not, but too cowardly to post before testing, purely based on the idea that:
```a := []int{}
fmt.Printf("%v\n", a[0:])
```
should not panic
guoger (Tue, 03 Jul 2018 03:44:04 GMT):
ok, but someone needs to explain this before I take my money out of pocket
kostas (Tue, 03 Jul 2018 03:48:50 GMT):
@guoger: When you *slice* (i.e. a[low:high]), the indices *can* go up to len(a).
kostas (Tue, 03 Jul 2018 03:48:57 GMT):
When you access an element, an index equal to len(a) is —of course— out of range and results in a run-time panic.
kostas (Tue, 03 Jul 2018 03:49:00 GMT):
In the snippet above we're slicing, so len(a) is a valid choice.
kostas (Tue, 03 Jul 2018 03:49:38 GMT):
The language spec could have been a bit clearer on this, though it does say somewhere in there:
kostas (Tue, 03 Jul 2018 03:49:44 GMT):
> ...the indices are in range if 0 <= low <= high <= len(a), otherwise they are out of range.
kostas (Tue, 03 Jul 2018 03:49:47 GMT):
https://golang.org/ref/spec#Slice_expressions
guoger (Tue, 03 Jul 2018 03:52:49 GMT):
hmm, slicing is half-open range, so in order to include the last element, `high` needs to be greater than last index, hence len()
guoger (Tue, 03 Jul 2018 03:53:09 GMT):
do you think this is the purpose?
kostas (Tue, 03 Jul 2018 03:53:12 GMT):
Not sure I follow?
guoger (Tue, 03 Jul 2018 04:51:29 GMT):
```
func main() {
a := []int{0,1,2}
fmt.Printf("%v\n", a[0:3])
fmt.Printf("%v\n", a[0:])
}
```
guoger (Tue, 03 Jul 2018 04:52:35 GMT):
I'm trying to understand the reason behind `high <= len`, instead of `high < len`. One reason I could think of, is that we need to be able to include the *last* element
guoger (Tue, 03 Jul 2018 04:52:59 GMT):
so we could write something in this code snippet
kostas (Tue, 03 Jul 2018 04:56:21 GMT):
Ah, I get you now. I'm not sure what the motivation is, but this seems likely.
suchith.arodi (Tue, 03 Jul 2018 18:25:37 GMT):
Has joined the channel.
guoger (Wed, 04 Jul 2018 05:36:44 GMT):
are we going to use ginkgo, gomega and counterfeiter for raft work?
kostas (Wed, 04 Jul 2018 09:13:43 GMT):
@guoger: Yes. If you look at https://github.com/hyperledger/fabric/tree/master/integration/nwo, it is demonstrating how the first two are used.
guoger (Wed, 04 Jul 2018 09:15:37 GMT):
cool, we used that in evm project as well :) although I hope we don't need to convert all existing test to use them for hygiene purpose
kostas (Wed, 04 Jul 2018 09:17:16 GMT):
We should be good, since the raft package is a clean slate.
guoger (Thu, 05 Jul 2018 10:26:48 GMT):
anyone knows how to subscribe to google doc, so I could get notified about new comments (not only replies)?
guoger (Thu, 05 Jul 2018 14:48:00 GMT):
@kostas is this reflected in design doc? my impression is that we'll be adding new gRPC service to facilitate message exchange among raft nodes, instead of accepting ingress tx. (in another word, stick with `broadcast` rpc)
https://jira.hyperledger.org/browse/FAB-9902?focusedCommentId=46333&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-46333
kostas (Thu, 05 Jul 2018 15:31:12 GMT):
@guoger: Treat the doc as the source of truth. (New RPC it is.)
kostas (Thu, 05 Jul 2018 15:31:52 GMT):
> anyone knows how to subscribe to google doc, so I could get notified about new comments (not only replies)?
I don't what I've done wrong in this doc, but I get no notifications of new comments and have to scan them manually.
kostas (Thu, 05 Jul 2018 16:12:12 GMT):
Screen Shot 2018-07-05 at 12.11.38.png
kostas (Thu, 05 Jul 2018 16:12:51 GMT):
@guoger: Clicking on the grey comment icon should give the options to choose to be notified of all comments. (But that still doesn't work for me.)
kostas (Thu, 05 Jul 2018 16:51:10 GMT):
Raft update:
kostas (Thu, 05 Jul 2018 16:51:15 GMT):
We've got the following outstanding items on the design --
1. https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHLQ
2. https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHOo
3. https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdGtU
kostas (Thu, 05 Jul 2018 16:51:15 GMT):
We've got the following outstanding items on the design:
1. https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHLQ
2. https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHOo
3. https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdGtU
kostas (Thu, 05 Jul 2018 16:51:15 GMT):
We've got the following outstanding items on the design:
1. What do we do if we _need_ to edit more than one certificates at once?
https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHLQ
2. Wire Raft RPC invocation to appropriate receiving FSM (remember that each channel
maps to a different cluster): https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHOo
3. Edit own consenter info: https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdGtU
kostas (Thu, 05 Jul 2018 16:51:15 GMT):
We've got the following outstanding items on the design:
1. What do we do if we _need_ to edit more than one certificates at once?
https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHLQ
2. ~Wire Raft RPC invocation to appropriate receiving FSM (remember that each channel
maps to a different cluster): https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHOo~
3. Edit own consenter info: https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdGtU
kostas (Thu, 05 Jul 2018 16:51:15 GMT):
We've got the following outstanding items on the design:
1. What do we do if we _need_ to edit more than one certificates at once?
https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHLQ
2. ~Wire Raft RPC invocation to appropriate receiving FSM (remember that each channel
maps to a different cluster):~ ~https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHOo~
3. Edit own consenter info: https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdGtU
kostas (Thu, 05 Jul 2018 16:51:15 GMT):
We've got the following outstanding items on the design:
1. What do we do if we _need_ to edit more than one certificates at once?
https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHLQ
2. ~Wire Raft RPC invocation to appropriate receiving FSM (remember that each channel
maps to a different cluster)~: ~https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHOo~
3. Edit own consenter info: https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdGtU
kostas (Thu, 05 Jul 2018 16:51:15 GMT):
We've got the following outstanding items on the design:
1. What do we do if we _need_ to edit more than one certificates at once?
https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHLQ
2. ~Wire Raft RPC invocation to appropriate receiving FSM~ (remember that each channel
maps to a different cluster): ~https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHOo~
3. Edit own consenter info: https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdGtU
kostas (Thu, 05 Jul 2018 16:51:15 GMT):
We've got the following outstanding items on the design:
1. What do we do if we _need_ to edit more than one certificates at once?
https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHLQ
2. ~Wire Raft RPC invocation to appropriate receiving FSM~ (~remember that each channel
maps to a different cluster~): ~https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHOo~
3. Edit own consenter info: https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdGtU
kostas (Thu, 05 Jul 2018 16:51:15 GMT):
We've got the following outstanding items on the design:
1. What do we do if we _need_ to edit more than one certificates at once?
https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHLQ
2. ~Wire Raft RPC invocation to appropriate receiving FSM~ (~remember that each channel
maps to a different cluster~): ~https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHOo~
3. Edit own consenter info: https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdGtU (@yacovm already has a proposal for this one)
kostas (Thu, 05 Jul 2018 16:51:15 GMT):
We've got the following outstanding items on the design:
1. What do we do if we _need_ to edit more than one certificates at once?
https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHLQ
2. ~Wire Raft RPC invocation to appropriate receiving FSM, remember that each channel
maps to a different cluster~: ~https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHOo~
3. Edit own consenter info: https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdGtU (@yacovm already has a proposal for this one)
kostas (Thu, 05 Jul 2018 16:51:15 GMT):
We've got the following outstanding items on the design:
1. What do we do if we _need_ to edit more than one certificates at once?
https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHLQ
2. ~Wire Raft RPC invocation to appropriate receiving FSM -- remember that each channel
maps to a different cluster~: ~https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHOo~
3. Edit own consenter info: https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdGtU (@yacovm already has a proposal for this one)
kostas (Thu, 05 Jul 2018 16:51:15 GMT):
We've got the following outstanding items on the design:
1. What do we do if we _need_ to edit more than one certificates at once?
https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHLQ
2. ~Wire Raft RPC invocation to appropriate receiving FSM~(~remember that each channel
maps to a different cluster~): ~https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHOo~
3. Edit own consenter info: https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdGtU (@yacovm already has a proposal for this one)
kostas (Thu, 05 Jul 2018 16:51:15 GMT):
We've got the following outstanding items on the design:
1. What do we do if we _need_ to edit more than one certificates at once?
https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHLQ
2. ~Wire Raft RPC invocation to appropriate receiving FSM.~ ~Remember that each channel
maps to a different cluster. ~https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHOo~
3. Edit own consenter info: https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdGtU (@yacovm already has a proposal for this one)
kostas (Thu, 05 Jul 2018 16:51:15 GMT):
We've got the following outstanding items on the design:
1. What do we do if we _need_ to edit more than one certificates at once?
https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHLQ
2. ~Wire Raft RPC invocation to appropriate receiving FSM~: ~https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHOo~
3. Edit own consenter info: https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdGtU (@yacovm already has a proposal for this one)
kostas (Thu, 05 Jul 2018 16:56:41 GMT):
I'm looking at 2 right now. If you have thoughts on any of these, post here or in the doc.
kostas (Thu, 05 Jul 2018 16:59:03 GMT):
If you have free cycles, and intend to contribute to the Raft work, studying the raftexample in the etcd repo https://github.com/coreos/etcd/tree/master/contrib/raftexample is a must.
kostas (Thu, 05 Jul 2018 17:03:09 GMT):
I've also taken a preliminary crack at breaking the work down into stories/weeks, but this will have to be edited further, given the outstanding items above: https://docs.google.com/spreadsheets/d/1R02-xjdl4hNw90kN4NZ1kX0f4QoIYn51_1UHlwy-GQs/edit
kostas (Thu, 05 Jul 2018 17:04:27 GMT):
If you see something that strikes you as wrong, post your counter-proposal, either here or in the doc.
kostas (Thu, 05 Jul 2018 17:20:28 GMT):
RE: 2 -- I'm seeing parallels to how the broadcast handler retrieves the chain corresponding to the requested channel by using the multichannel registrar: https://github.com/hyperledger/fabric/blob/77c3aa6ce5b0cfba93bfda009095886dbcadff91/orderer/common/broadcast/broadcast.go#L93
kostas (Thu, 05 Jul 2018 17:20:28 GMT):
RE: 2 -- I'm seeing parallels to how the broadcast handler retrieves the chain (technically: [ChainSupport object](https://github.com/hyperledger/fabric/blob/77c3aa6ce5b0cfba93bfda009095886dbcadff91/orderer/common/multichannel/chainsupport.go#L22))corresponding to the requested channel by using the multichannel registrar: https://github.com/hyperledger/fabric/blob/77c3aa6ce5b0cfba93bfda009095886dbcadff91/orderer/common/broadcast/broadcast.go#L93
kostas (Thu, 05 Jul 2018 17:20:28 GMT):
RE: 2 -- I'm seeing parallels to how the broadcast handler retrieves the chain (technically: [ChainSupport object](https://github.com/hyperledger/fabric/blob/77c3aa6ce5b0cfba93bfda009095886dbcadff91/orderer/common/multichannel/chainsupport.go#L22)) corresponding to the requested channel by using the multichannel registrar: https://github.com/hyperledger/fabric/blob/77c3aa6ce5b0cfba93bfda009095886dbcadff91/orderer/common/broadcast/broadcast.go#L93
kostas (Thu, 05 Jul 2018 17:21:04 GMT):
If out EtcdRaftProposal server handler has access to that registrar we can proceed accordingly, but we're not done yet.
kostas (Thu, 05 Jul 2018 17:21:04 GMT):
If our EtcdRaftProposal server handler has access to that registrar we can proceed accordingly, but we're not done yet.
kostas (Thu, 05 Jul 2018 17:25:15 GMT):
The question is: which method do we invoke on that `ChainSupport` object in order to route the transaction to the right Raft FSM.
kostas (Thu, 05 Jul 2018 17:26:26 GMT):
As I see it, we either add a new method to the already big [Chain interface](https://github.com/hyperledger/fabric/blob/release-1.2/orderer/consensus/consensus.go#L34), or we re-use one of its methods, if the semantics fit.
kostas (Thu, 05 Jul 2018 17:29:14 GMT):
(And I don't see any method that could be re-used for it.)
jyellick (Thu, 05 Jul 2018 17:30:40 GMT):
Having the consenter reach into the registrar feels wrong to me. I would have expected that when the consenter object registers the RPC service, that it would setup and handle routing to the correct FSM.
kostas (Thu, 05 Jul 2018 17:33:27 GMT):
We _can_ establish a similar sort of mapping to `r.chains`, and that gives us the freedom to route w/ piggybacking on any of the Fabric interfaces. Still not sold on why this wouldn't be a query on the registrar though.
kostas (Thu, 05 Jul 2018 17:33:27 GMT):
We _can_ establish a similar mapping to `r.chains` on the plugin side of things, and that gives us the freedom to route w/ piggybacking on any of the Fabric interfaces. Still not sold on why this wouldn't be a query on the registrar though.
jyellick (Thu, 05 Jul 2018 17:37:02 GMT):
Do you mean "a mapping similar to `r.chains`" or, "a similar structure which maps into `r.chains`"?
kostas (Thu, 05 Jul 2018 17:38:56 GMT):
I meant the former. But to keep things simpler: a map whose key is the channel or cluster ID, and the value is whatever object we (Raft plugin authors) wish to expose to the EtcdRaftProposal server handler.
kostas (Thu, 05 Jul 2018 17:38:56 GMT):
I meant the former. But to keep things simpler: a map whose key is the channel or cluster ID, and the value is a reference to whatever object we (Raft plugin authors) wish to expose to the EtcdRaftProposal server handler.
kostas (Thu, 05 Jul 2018 17:39:25 GMT):
That wires through to the Raft FSM corresponding to the channel/cluster.
jyellick (Thu, 05 Jul 2018 17:43:57 GMT):
Got it, thanks. My inclination then is to maintain this mapping separately. The most pragmatic reason which jumps to mind is that the `Chain` is not registered with the registrar until after the consenter starts, so there would likely be a race or deadlock while attempting to route the initial handshake messages.
jyellick (Thu, 05 Jul 2018 17:44:49 GMT):
In short, if you wish to rely on the routing working during the `HandleChain` invocation, then I think there will be problems.
kostas (Thu, 05 Jul 2018 17:50:25 GMT):
> so there would likely be a race or deadlock while attempting to route the initial handshake messages.
If key is not found, just fail the call?
jyellick (Thu, 05 Jul 2018 17:58:57 GMT):
Reflecting on it a bit more, you are certainly right, as channel creation is not a synchronous operation, even with a separate mapping, we will have to contend with messages which are received for a channel we do not (yet) know about. I'm not sure what the right way to handle that is, because warning about unknown channels will be inelegant, but indefinitely buffering would also be a problem. I expect we would want to buffer the messages up to some total size and perhaps duration before logging an error and or dropping.
Still, I would envision that `HandleChain` starts a go routine which begins sending a receiving RPC messages. The sending bit seems more problematic to me. Assuming that the routing is done externally based on a mapping, then you would have to contend with that mapping not existing in the case of re-using the registrar. Considering that the mapping is simple to code and low overhead, it seems simpler than dealing with failures on outgoing messages to me.
kostas (Thu, 05 Jul 2018 18:15:45 GMT):
> Assuming that the routing is done externally based on a mapping, then you would have to contend with that mapping not existing in the case of re-using the registrar.
So something still doesn't stick here which is why I'm challenging this. (Other than, I'm not married to either idea, and yeah, the external mapping is simple enough and better in that we don't have to mess around with existing interfaces, that we should go for it.) Namely, you need a Raft FSM object in order to populate that external mapping, and you can't have a Raft FSM object unless you instantiate a chain object. And that chain object is first registered on the registrar.
kostas (Thu, 05 Jul 2018 18:16:26 GMT):
The TL;DR version is that the registrar will always get populated first.
kostas (Thu, 05 Jul 2018 18:16:26 GMT):
The TL;DR version is that the registrar mapping will always get populated first.
jyellick (Thu, 05 Jul 2018 18:23:08 GMT):
Naturally, while typing up a long example, I think I see the error in my thinking
jyellick (Thu, 05 Jul 2018 18:23:24 GMT):
I honestly forgot that there was a `Start()` function which is invoked after the registration takes place.
jyellick (Thu, 05 Jul 2018 18:23:42 GMT):
I was assuming it was the `HandleChain` which kicked off the service go routine, but that should not be the case.
jyellick (Thu, 05 Jul 2018 18:24:59 GMT):
It's relatively trivial to fix, but the registrar map is not actually memory order safe. Meaning, you are guaranteed to get some consistent view of the registry, but not necessarily the current one (as there is no locking, and updating the map is done through a non-atomic pointer swap)
kostas (Thu, 05 Jul 2018 18:25:41 GMT):
The newMap = copy of old Map and then swap the maps bit, right?
jyellick (Thu, 05 Jul 2018 18:25:48 GMT):
Correct
kostas (Thu, 05 Jul 2018 18:25:49 GMT):
Was looking at that earlier today.
jyellick (Thu, 05 Jul 2018 18:27:53 GMT):
I was probably trying to be too clever, but at the time, I could not come up with any reason why the map needed to be memory safe (as the broadcast/deliver clients were already asynchronous, a transient failure didn't matter)
kostas (Thu, 05 Jul 2018 18:39:53 GMT):
Still, do you agree that there is no way in which the external mapping can have an edge over the registrar mapping when it comes to being aware of instantiated channels?
jyellick (Thu, 05 Jul 2018 18:44:00 GMT):
It can have knowledge _slightly_ sooner, but not enough to be significant.
kostas (Thu, 05 Jul 2018 18:44:35 GMT):
Not disputing this, but disputing this:
kostas (Thu, 05 Jul 2018 18:44:37 GMT):
> then you would have to contend with that mapping not existing in the case of re-using the registrar
jyellick (Thu, 05 Jul 2018 18:45:05 GMT):
Ah, that statement was done under the assumption that the servicing go routine was spawned during `HandleChain` and not during `Start`
jyellick (Thu, 05 Jul 2018 18:45:14 GMT):
So, scratch that assertion
kostas (Thu, 05 Jul 2018 18:45:15 GMT):
Negligible advantage or not (and assuming the lack of atomic swap not being an issue), registrar's mapping comes first.
jyellick (Thu, 05 Jul 2018 18:45:25 GMT):
I disagree there
kostas (Thu, 05 Jul 2018 18:45:37 GMT):
How so?
jyellick (Thu, 05 Jul 2018 18:45:45 GMT):
`HandleChain` is called before the registrar map key is created.
jyellick (Thu, 05 Jul 2018 18:46:44 GMT):
The order is:
```chain, _ := HandleChain
registrar.chains[channelID] = chain
chain.start
```
jyellick (Thu, 05 Jul 2018 18:46:44 GMT):
The order is:
```chain, _ := HandleChain(...)
registrar.chains[channelID] = chain
chain.start
```
jyellick (Thu, 05 Jul 2018 18:46:44 GMT):
The order is:
```chain, _ := HandleChain(...)
registrar.chains[channelID] = chain
chain.start()
```
jyellick (Thu, 05 Jul 2018 18:46:44 GMT):
The order is:
```chain, _ := consenter.HandleChain(...)
registrar.chains[channelID] = chain
chain.start()
```
kostas (Thu, 05 Jul 2018 18:47:34 GMT):
Got it, and you're saying -- if you populate your external map on the handlechain invocation, the edge doesn't apply.
jyellick (Thu, 05 Jul 2018 18:47:39 GMT):
Correct
kostas (Thu, 05 Jul 2018 18:48:15 GMT):
Right, though the FSM should be spinned up on the start call, so the edge is lost.
kostas (Thu, 05 Jul 2018 18:48:32 GMT):
At any rate, I know get why you made that claim.
kostas (Thu, 05 Jul 2018 18:48:32 GMT):
At any rate, I now get why you made that claim.
kostas (Thu, 05 Jul 2018 18:48:32 GMT):
At any rate, I get why you made that claim.
C0rWin (Thu, 05 Jul 2018 19:37:36 GMT):
Has joined the channel.
kostas (Fri, 06 Jul 2018 00:54:05 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=7EzdcBFb9q3tp7jCh
kostas (Fri, 06 Jul 2018 00:54:29 GMT):
The only outstanding one is 1. I'll give that some thought tomorrow.
kostas (Fri, 06 Jul 2018 00:54:29 GMT):
At the time of this writing, the only outstanding one is 1. I'll give that some thought tomorrow.
kostas (Fri, 06 Jul 2018 00:54:29 GMT):
At the time of this writing, the only outstanding one is 1.
kostas (Fri, 06 Jul 2018 00:54:29 GMT):
At the time of this writing, the only outstanding one is item number 1.
kostas (Fri, 06 Jul 2018 00:54:29 GMT):
At the time of this writing, the only outstanding one is the first item in that list.
guoger (Fri, 06 Jul 2018 02:53:41 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=exWSi97vjGPqSiDdt) @kostas mind elaborate a bit?
yacovm (Fri, 06 Jul 2018 07:03:19 GMT):
@kostas we have another item that is - I think we should not change the semantics of the successful Broadcast() call (and if it is successful - it is in some block)
yacovm (Fri, 06 Jul 2018 07:04:18 GMT):
I think we should delay the broadcast until the block is cut and the transaction is in, or a timeout expires
yacovm (Fri, 06 Jul 2018 07:07:43 GMT):
https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAAB62Ch5M
yacovm (Fri, 06 Jul 2018 07:09:11 GMT):
(Linked to Jason's comment) - Do you want to discuss it? @guoger @C0rWin @kostas @jyellick
guoger (Fri, 06 Jul 2018 07:36:26 GMT):
I tend to agree with @yacovm that we should avoid changing semantics of return code, e.g. couldn't we add `202 Accept` to make it more precise? And since our `Broadcast` is already bidi stream, we could still send `200 OK` to client when tx is eventually included into block. Also, could you elaborate on shared queue solution? I couldn't immediately figure out how that solves our problem
guoger (Fri, 06 Jul 2018 07:50:34 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=gnnvTL2LBs6D7uZxg) also, would it be a problem if a slower orderer node, which has not instantiate newly created channel yet, receives raft messages targeting this channel and error out?
jyellick (Fri, 06 Jul 2018 13:36:59 GMT):
> I think we should not change the semantics of the successful Broadcast() call (and if it is successful - it is in some block)
This is not exactly the existing semantics. To be precise, the current semantics promise a client that: "If you receive a 200 back, then the message has entered consensus and will not be discarded due to a crash. If the message is still valid once ordered, then it will commit in a block." Note particularly that if a client's access is revoked after a 200 but before the message receives total order, then the message is simply discarded.
yacovm (Fri, 06 Jul 2018 13:38:27 GMT):
what I'm saying is - the SDK has a certain logic according to which it operates
yacovm (Fri, 06 Jul 2018 13:38:50 GMT):
if we say we provide weaker guarantees then we must express that somehow in the API so the SDK can distinguish, no?
yacovm (Fri, 06 Jul 2018 13:39:06 GMT):
the alternative would be to not change the API and provide the same guarantees
jyellick (Fri, 06 Jul 2018 13:40:55 GMT):
As we move towards BFT I think it is unavoidable that we weaken the guarantees. A client will have to wait for the request to commit and if it does not, resubmit it to another node.
jyellick (Fri, 06 Jul 2018 13:41:36 GMT):
Even if we required that the client submit to f+1 replicas, the client generally still must acknowledge the possibility of failure.
kostas (Fri, 06 Jul 2018 13:41:46 GMT):
^^ This.
kostas (Fri, 06 Jul 2018 13:42:13 GMT):
As I wrote in a comment in the doc, there will always be a case that the client has to resubmit.
kostas (Fri, 06 Jul 2018 13:43:19 GMT):
So I'm wary of building additional logic to the orderer that will increase our surface, and may not even address the problem fully (in whatever bizzare, BFT scenario comes up next).
yacovm (Fri, 06 Jul 2018 13:44:45 GMT):
all right then the question that remains is - whether we should return something else than `Status.Success` or not?
yacovm (Fri, 06 Jul 2018 13:45:07 GMT):
perhaps it's not worth to do that and we can just say that if the OSN crashes
yacovm (Fri, 06 Jul 2018 13:45:19 GMT):
then the SDK should have some timeout built in
yacovm (Fri, 06 Jul 2018 13:45:36 GMT):
that assumes that if after `K` blocks the event hub didn't notify about that txn
yacovm (Fri, 06 Jul 2018 13:45:44 GMT):
then something bad happened
yacovm (Fri, 06 Jul 2018 13:45:46 GMT):
?
kostas (Fri, 06 Jul 2018 13:46:39 GMT):
So, the (perhaps naive) way I was thinking about it is: success == the OSN got it and it'll attempt to order it, and that's it.
kostas (Fri, 06 Jul 2018 13:47:05 GMT):
The SDK should have a timer built-in, as you suggest, and if it doesn't see it in K blocks, it retries.
kostas (Fri, 06 Jul 2018 13:47:39 GMT):
I think we're saying the same thing, it's just this phrase that threw me off, which is why I repeated everything above:
kostas (Fri, 06 Jul 2018 13:47:43 GMT):
> perhaps it's not worth to do that and we can just say that if the OSN crashes
kostas (Fri, 06 Jul 2018 13:49:51 GMT):
I've been thinking about point #1 and I think I see a bigger issue.
yacovm (Fri, 06 Jul 2018 13:50:01 GMT):
what's that?
yacovm (Fri, 06 Jul 2018 13:50:07 GMT):
(what's point 1? )
kostas (Fri, 06 Jul 2018 13:50:15 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=HGNko56ASfPsHA4mu
yacovm (Fri, 06 Jul 2018 13:50:19 GMT):
ah
kostas (Fri, 06 Jul 2018 13:51:02 GMT):
So, for safety reasons, the reconfiguraton in Raft is a two-phase process.
kostas (Fri, 06 Jul 2018 13:52:26 GMT):
You temporarily migrate to a joint-consensus state between old_conf and new_conf before switching to new_conf, and you need a majority of old_conf and new_conf to agree.
kostas (Fri, 06 Jul 2018 13:52:56 GMT):
(I am waving my hands over this at the moment, I know. Bear with me for a sec.)
kostas (Fri, 06 Jul 2018 13:53:22 GMT):
What we've proposed in the doc is that the Fabric configuration change comes first, and then the Raft one.
kostas (Fri, 06 Jul 2018 13:53:50 GMT):
Which means we cut off read/write access to part of old_conf right away.
kostas (Fri, 06 Jul 2018 13:54:02 GMT):
And I am concerned about what implications this may have.
kostas (Fri, 06 Jul 2018 13:54:24 GMT):
So, I need to go over this part again.
yacovm (Fri, 06 Jul 2018 13:55:12 GMT):
so maybe we can do something else?
kostas (Fri, 06 Jul 2018 13:55:21 GMT):
(I think that since we change one node at a time we should be good?)
kostas (Fri, 06 Jul 2018 13:55:28 GMT):
> so maybe we can do something else?
Shoot, I'm all ears.
yacovm (Fri, 06 Jul 2018 13:55:39 GMT):
what if we step back for a second and design that membership table in another way
yacovm (Fri, 06 Jul 2018 13:55:51 GMT):
to be more "raft"-friendly
yacovm (Fri, 06 Jul 2018 13:56:05 GMT):
put the integers that denote the IDs as real IDs inside the table
yacovm (Fri, 06 Jul 2018 13:56:28 GMT):
that's the "left most column" ;)
yacovm (Fri, 06 Jul 2018 13:56:39 GMT):
the other columns are the endpoints, and certificates and MSP IDs
yacovm (Fri, 06 Jul 2018 13:57:00 GMT):
so, now - to change the certificate(s) of the `n` node you don't need a new node for raft
yacovm (Fri, 06 Jul 2018 13:57:12 GMT):
you just change it for fabric and Raft doesn't care about it
yacovm (Fri, 06 Jul 2018 13:57:33 GMT):
when you add a new node - you just add a new entry to the table, similar when removing nodes
yacovm (Fri, 06 Jul 2018 13:57:45 GMT):
and there is another config value that tracks that sequences aren't re-used
yacovm (Fri, 06 Jul 2018 13:58:01 GMT):
like the one you had - the "next sequence"
kostas (Fri, 06 Jul 2018 13:58:41 GMT):
I'm slow, can you give me an example?
kostas (Fri, 06 Jul 2018 13:58:50 GMT):
I will note that we've agreed to do this per your suggestion: https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHLA
kostas (Fri, 06 Jul 2018 13:59:00 GMT):
(MSP field to allow editing.)
yacovm (Fri, 06 Jul 2018 13:59:30 GMT):
ID | endpoint | TLSclientCert | TLSserverCert | MSP_ID
-------------------------------------------------------
yacovm (Fri, 06 Jul 2018 13:59:30 GMT):
ID | endpoint | TLSclientCert | TLSserverCert | MSP_ID
-------------------------------------------------------
0 | osn0.org1 | ..... | .............. | ORG1MSP
yacovm (Fri, 06 Jul 2018 13:59:30 GMT):
ID | endpoint | TLSclientCert | TLSserverCert | MSP_ID
-------------------------------------------------------
1 | osn0.org1 | ..... | .............. | ORG1MSP
yacovm (Fri, 06 Jul 2018 13:59:30 GMT):
ID | endpoint | TLSclientCert | TLSserverCert | MSP_ID
-------------------------------------------------------
1 | osn0.org1 | ............... | .............. | ORG1MSP
kostas (Fri, 06 Jul 2018 14:01:00 GMT):
> put the integers that denote the IDs as real IDs inside the table
kostas (Fri, 06 Jul 2018 14:01:11 GMT):
(Note sure what this means?)
yacovm (Fri, 06 Jul 2018 14:02:12 GMT):
edited above
yacovm (Fri, 06 Jul 2018 14:02:21 GMT):
so, the raft IDs are going to be in that table
yacovm (Fri, 06 Jul 2018 14:02:50 GMT):
so when you do a certificate refresh/rotation you never need a raft membership change
yacovm (Fri, 06 Jul 2018 14:02:54 GMT):
just a fabric one
kostas (Fri, 06 Jul 2018 14:05:28 GMT):
That is a good idea. However, what is the exact problem we're solving here?
yacovm (Fri, 06 Jul 2018 14:05:51 GMT):
the problem is that most reconfigs are not going to be additions or removal of actual nodes
yacovm (Fri, 06 Jul 2018 14:05:58 GMT):
they're going to be certificate renewels
yacovm (Fri, 06 Jul 2018 14:05:58 GMT):
they're going to be certificate renewals
yacovm (Fri, 06 Jul 2018 14:06:28 GMT):
so if we do this we don't need a raft reconfig
kostas (Fri, 06 Jul 2018 14:06:52 GMT):
Excellent. I just want to stress this, going back to your #1 point in the list above --
kostas (Fri, 06 Jul 2018 14:07:44 GMT):
If we have to deal with a *concurrent* change of more than half of the certs, then all bets are off right? It's a fault outside our tolerance levels.
yacovm (Fri, 06 Jul 2018 14:09:05 GMT):
why?
yacovm (Fri, 06 Jul 2018 14:09:52 GMT):
changing more than half of the certs now (in the table above) doesn't require consensus
kostas (Fri, 06 Jul 2018 14:10:18 GMT):
Yes! I was recognizing the error in my logic, as soon as I started typing my explanation.
yacovm (Fri, 06 Jul 2018 14:10:40 GMT):
I'm also not accurate
yacovm (Fri, 06 Jul 2018 14:10:48 GMT):
it requires consensus ;)
yacovm (Fri, 06 Jul 2018 14:10:53 GMT):
but you know what i meant
kostas (Fri, 06 Jul 2018 14:10:59 GMT):
Yes, yes.
kostas (Fri, 06 Jul 2018 14:11:03 GMT):
So, then your suggestion here https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdHLA covers point #1 ?
kostas (Fri, 06 Jul 2018 14:11:18 GMT):
Or is there a subtle change between what you suggested here vs what you wrote there?
yacovm (Fri, 06 Jul 2018 14:12:12 GMT):
we still haven't solved your original problem though have we?
yacovm (Fri, 06 Jul 2018 14:12:40 GMT):
if you do a membership change - say, an expansion it's going to happen in 2 steps
kostas (Fri, 06 Jul 2018 14:12:40 GMT):
Right, will look into this now. As I wrote afterwards, I _think_ we should be good, since we're progressing one node at a time.
kostas (Fri, 06 Jul 2018 14:13:14 GMT):
So we should be good to get a majority. But I'm waving my hands over the exact sequence because I've frankly forgotten the details of it.
yacovm (Fri, 06 Jul 2018 14:13:34 GMT):
hmmm so question - what if we do a membership expansion - and a config block is cut that authorizes the new OSNs and then someone tries to submit a transaction before the cluster has time to reconfigure
yacovm (Fri, 06 Jul 2018 14:13:39 GMT):
how does that work? I'm just curious
kostas (Fri, 06 Jul 2018 14:14:22 GMT):
So, Fabric config change adds OSN 4 to the mix.
kostas (Fri, 06 Jul 2018 14:14:35 GMT):
But Raft doesn't see OSN 4 as part of its cluster yet.
kostas (Fri, 06 Jul 2018 14:15:21 GMT):
In this case I expect the transaction to fail.
kostas (Fri, 06 Jul 2018 14:16:33 GMT):
And then resubmission after K blocks, as we've discussed above, should take care of it.
kostas (Fri, 06 Jul 2018 14:17:22 GMT):
If this becomes an issue in production, then it's queue + retry X times before we give up, along the discussion we had in the doc yesterday.
kostas (Fri, 06 Jul 2018 14:17:24 GMT):
WDYT?
yacovm (Fri, 06 Jul 2018 14:18:04 GMT):
makes sense to me
kostas (Fri, 06 Jul 2018 14:18:52 GMT):
This issue would be resolved if you were to post the Raft change before the Fabric config change, but that opens a new can of worms. (Mainly that you can derive the Raft change from the Fabric config change, but not vice-versa.)
yacovm (Fri, 06 Jul 2018 14:33:04 GMT):
so we can say for instructions that you should never:
1) Replace an existing node's endpoints/certificate(s) while adding/removing nodes
2) Remove/add more than 1 node at a time
@kostas wdyt?
yacovm (Fri, 06 Jul 2018 14:33:04 GMT):
so we can say for instructions that you should never:
1) Replace an existing node's endpoints/certificate(s) while adding/removing nodes
2) Remove/add more than 1 node at a time
@kostas wdyt?
kostas (Fri, 06 Jul 2018 14:33:48 GMT):
That sounds right to me.
kostas (Fri, 06 Jul 2018 14:34:46 GMT):
Given that we won't pipeline config updates (this is listed in the doc), adding the logic that prevents adding/removing more than 1 node at a time is all we need, and (1) will automatically take care of itself.
jeroiraz (Fri, 06 Jul 2018 21:04:35 GMT):
Has joined the channel.
kostas (Sat, 07 Jul 2018 03:52:20 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=JQsw9GQNJaf5C7pwi
kostas (Sat, 07 Jul 2018 03:52:35 GMT):
It's probably the fact that it's late but I'm second myself I guess here.
kostas (Sat, 07 Jul 2018 03:52:35 GMT):
It's probably the fact that it's late but I'm second-guessing myself on this statement ^^.
kostas (Sat, 07 Jul 2018 03:52:35 GMT):
It's probably the fact that it's late, but I'm second-guessing myself on this statement ^^.
kostas (Sat, 07 Jul 2018 03:53:00 GMT):
And perhaps there is some truth to this one:
kostas (Sat, 07 Jul 2018 03:53:03 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=wjbCgxniGfMi3B7mp
kostas (Sat, 07 Jul 2018 03:53:28 GMT):
Let's work with an example.
kostas (Sat, 07 Jul 2018 03:53:28 GMT):
And perhaps there is some truth to this statement ^^. Let's work with an example.
kostas (Sat, 07 Jul 2018 03:53:42 GMT):
5 replicas in the Raft cluster, i.e. 5 OSNs.
kostas (Sat, 07 Jul 2018 03:54:40 GMT):
If 3 of those replicas have their TLS expired, we cannot commit the Fabric configuration change that will modify their TLS entries in the "consenters" field.
kostas (Sat, 07 Jul 2018 03:56:03 GMT):
So I don't think we've actually solved the problem in point 1.
kostas (Sat, 07 Jul 2018 03:57:13 GMT):
But, as I wrote earlier today, this is fine, in that it constitutes a failure above the protocol's fault tolerance levels.
kostas (Sat, 07 Jul 2018 03:57:13 GMT):
But, as I wrote (temporarily) earlier today, this is fine, in that it constitutes a failure above the protocol's fault tolerance levels.
kostas (Sat, 07 Jul 2018 03:57:13 GMT):
But, as I wrote (temporarily) earlier today, I _think_ this should be deemed acceptable, in that it constitutes a failure above the protocol's fault tolerance levels.
kostas (Sat, 07 Jul 2018 03:57:25 GMT):
So we should not spend any cycles in trying to address it.
kostas (Sat, 07 Jul 2018 03:57:25 GMT):
This is a side-effect of this joining we have going on at the hip with the TLS certificates, but any other way of assigning IDs to Raft nodes (which is part of we're achieving with the TLS binding) seems too brittle (or more brittle anyway) to me.
kostas (Sat, 07 Jul 2018 03:57:25 GMT):
This is a side-effect of this joining we have going on at the hip with the TLS certificates, but any other way of assigning IDs to Raft nodes (which is part of we're achieving with the TLS binding) seems too brittle (or more brittle anyway) to me. Please let me know your thoughts.
kostas (Sat, 07 Jul 2018 03:57:25 GMT):
This is due to the fact that the OSN certificates are part of the global (vs local) configuration, and as such, any updates to them be have to be ordered. In short, our "use the TLS certificates to find the Raft ID" scheme aside, I think we'd bump into the exact same issue with say, the BFT-based ordering service, as long as information about the OSN's TLS certificates is persisted on the chain (as opposed to, say, `orderer.yaml`.) So I view this as an issue that we can do nothing about. Let me know your thoughts.
kostas (Sat, 07 Jul 2018 03:58:34 GMT):
As for this:
kostas (Sat, 07 Jul 2018 03:58:35 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=G2j7YjqRESWHMri29
kostas (Sat, 07 Jul 2018 03:59:26 GMT):
Recall that the issue I was concerned with was:
kostas (Sat, 07 Jul 2018 03:59:44 GMT):
We wish to remove (or add) a Raft replica.
kostas (Sat, 07 Jul 2018 04:00:10 GMT):
When the Fabric configuration change takes effect, that replica's read/write access is cut off immediately.
kostas (Sat, 07 Jul 2018 04:01:01 GMT):
Then we follow up with the Raft configuration change.
kostas (Sat, 07 Jul 2018 04:01:41 GMT):
And Raft operates in a 2-phase approach in this case:
kostas (Sat, 07 Jul 2018 04:01:41 GMT):
And Raft operates in a 2-phase approach in this case in which the node we wish to, say, remove doesn't actually gets removed right away:
kostas (Sat, 07 Jul 2018 04:03:10 GMT):
The leader immediately _applies_ and operates under a configuration where it seeks a majority from both old_conf and new_conf.
kostas (Sat, 07 Jul 2018 04:03:52 GMT):
When it manages to get that old_conf + new_conf (joint consensus) entry _committed_, it then proceeds with attempting to commit new_conf.
kostas (Sat, 07 Jul 2018 04:06:30 GMT):
For us, the fact that we're only allowing one node to be added/removed at a time, means that during the joint consensus phase, we can get a majority vote even with kicking the removed node off the network right away. So we can make progress.
kostas (Sat, 07 Jul 2018 04:06:58 GMT):
So there goes that as well. (Unless I'm missing something, in which case, please let me know.)
kostas (Sat, 07 Jul 2018 04:06:58 GMT):
So there goes that one.
yacovm (Sat, 07 Jul 2018 06:37:58 GMT):
well but @kostas when I said it'll work I meant as in response to what you said:
> If we have to deal with a *concurrent* change of more than half of the certs, then all bets are off right? It's a fault outside our tolerance levels.
As for:
> If 3 of those replicas have their TLS expired, we cannot commit the Fabric configuration change that will modify their TLS entries in the "consenters" field.
when TLS certificates expire, you don't close the TLS connection... so the nodes can still communicate if they expired as long as the connection didn't break.
yacovm (Sat, 07 Jul 2018 06:37:58 GMT):
well but @kostas when I said it'll work I meant as in response to what you said:
> If we have to deal with a *concurrent* change of more than half of the certs, then all bets are off right? It's a fault outside our tolerance levels.
There is nothing in the *concurrent* change that breaks if you go with my idea of the IDs inside the mapping, because it doesn't affect the raft config, just the fabric config.
As for:
> If 3 of those replicas have their TLS expired, we cannot commit the Fabric configuration change that will modify their TLS entries in the "consenters" field.
when TLS certificates expire, you don't close the TLS connection... so the nodes can still communicate if they expired as long as the connection didn't break.
yacovm (Sat, 07 Jul 2018 06:46:05 GMT):
Of course that if the connections were closed then we're disconnected and dead in the water ;)
ishakboyaci (Sun, 08 Jul 2018 22:33:32 GMT):
Has joined the channel.
kostas (Mon, 09 Jul 2018 02:22:51 GMT):
> There is nothing in the *concurrent* change that breaks if you go with my idea of the IDs inside the mapping, because it doesn't affect the raft config, just the fabric config.
Hm, not sure I follow. Can you walk me through an example where we'd be stuck without your modification, and then show how the suggested modification saves us?
kostas (Mon, 09 Jul 2018 02:27:40 GMT):
If we clear this, we can lock on the current design I think. (At least until the next issue comes up.)
yacovm (Mon, 09 Jul 2018 07:22:45 GMT):
@kostas you can't do a concurrent change in the original way because you are supposed to only update 1 raft node at a time in small clusters, so in the original way
yacovm (Mon, 09 Jul 2018 11:10:47 GMT):
so I looked at the raft sample in the `etcd/contrib/raftexample` @kostas and I think we need to elaborate more in the design doc about all the raft plumbing stuff
yacovm (Mon, 09 Jul 2018 11:10:55 GMT):
the fabric plumbing is very elaborate which is good
yacovm (Mon, 09 Jul 2018 11:11:02 GMT):
but i think the raft plumbing isn't elaborated enough IMO
yacovm (Mon, 09 Jul 2018 11:11:45 GMT):
if we can have pseudo code in the document based on the sample - that would help make the document easier to review and for us to ingest (though I guess you might already have, but I haven't yet)
yacovm (Mon, 09 Jul 2018 12:33:04 GMT):
also 1 more thing I'm missing - all OSNs need to have the entire ledger since they serve Deliver requests.
Obviously we can't use a snapshot for that because the data is huge so the OSN that joins would need to call Deliver on its own, right?
But at the same time - it may serve Deliver() requests from peers for blocks it doesn't have
yacovm (Mon, 09 Jul 2018 12:33:04 GMT):
also 1 more thing I'm missing - all OSNs need to have the entire ledger since they serve Deliver requests.
Obviously we can't use a snapshot for that because the data is huge so the OSN that joins would need to call Deliver on its own, right?
But at the same time - it may serve Deliver() requests from peers for blocks it doesn't have. I guess we would need to return `SERVICE_UNAVAILABLE` ?
yacovm (Mon, 09 Jul 2018 12:36:50 GMT):
the doc says:
> Once R1 has received block 180, L should invoke the ReportSnapshot
but I think it's sub-optimal because while we sync the blocks in the ledger, the node doesn't sync with the raft cluster, right?
Can we reverse the order?
When a node joins - it immediately requests a snapshot, and in the background it sync the other blocks
yacovm (Mon, 09 Jul 2018 12:51:47 GMT):
this would, however - require that the leader would validate the incoming broadcasts from the new node because it can't do that instead as it didn't get all the config blocks.
yacovm (Mon, 09 Jul 2018 12:52:21 GMT):
@jyellick is it possible to instantiate an OSN with latest config block? :thinking:
jyellick (Mon, 09 Jul 2018 13:41:44 GMT):
> @jyellick is it possible to instantiate an OSN with latest config block? 🤔
@yacovm It would be relatively trivial to bootstrap the orderer with the latest config block (for each channel). There would certainly need to be a few things modified, particularly some of the ledger interfaces, but in general, the orderer code is designed to accept/expect that there may be some truncation of the blockchains at config blocks.
yacovm (Mon, 09 Jul 2018 13:42:34 GMT):
So let me understand something... when a follower gets an envelope, it forwards it to the leader.
Obviously we'd want the follower to validate the envelope, but for that we need to attach to the validation - the config block at which the follower got the message.
This is problematic for followers that are added to the cluster when the cluster expands, from obvious reasons
yacovm (Mon, 09 Jul 2018 13:42:48 GMT):
such a follower would validate the messages for nothing because it will be re-validated by the leader anyway
yacovm (Mon, 09 Jul 2018 13:43:09 GMT):
so I'm wondering if we can somehow make the follower spawn from the latest config block
yacovm (Mon, 09 Jul 2018 13:43:17 GMT):
and thus - it will validate everything correctly, no?
jyellick (Mon, 09 Jul 2018 13:43:39 GMT):
In Kafka, we simply attach the config sequence at which the tx was validated.
jyellick (Mon, 09 Jul 2018 13:43:59 GMT):
If it matches the leader's, then the leader may accept, if it is lower, then the leader must revalidate, if it is higher, this would be a byzantine fault.
yacovm (Mon, 09 Jul 2018 13:44:30 GMT):
ok and what about:
yacovm (Mon, 09 Jul 2018 13:44:42 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=bnr6acnAkpHcCxyZd) this? ^
yacovm (Mon, 09 Jul 2018 13:44:42 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=bnr6acnAkpHcCxyZd) this? \/
yacovm (Mon, 09 Jul 2018 13:44:42 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=bnr6acnAkpHcCxyZd) this? :arrow_down:
jyellick (Mon, 09 Jul 2018 13:46:36 GMT):
This is a better question for @kostas but my impression was that the snapshoting was done actively by the leader to allow the followers to garbage collect the log. So, if a follower joins late, and its windows do not align with the cluster, it will request the snapshot, sync the blocks, and then begin participating.
jyellick (Mon, 09 Jul 2018 13:47:15 GMT):
I considered for quite a while whether we should require that the node complete syncing before participating in the cluster, as it's not strictly necessary that the sync complete, so long as the snapshot contains the latest config block, and a hash of the previous block header.
jyellick (Mon, 09 Jul 2018 13:48:05 GMT):
However, this would require even more modifications to the ledger, and the failure cases become much more complex. Until we have a more compelling reason to do so, I think halting the follower's participation in consensus until it has completely caught up is the best solution.
yacovm (Mon, 09 Jul 2018 13:48:28 GMT):
why modifications to the ledger?
jyellick (Mon, 09 Jul 2018 13:48:49 GMT):
The ledger block storage format is an binary log of blocks
jyellick (Mon, 09 Jul 2018 13:48:56 GMT):
It does not support holes or out of order commit.
yacovm (Mon, 09 Jul 2018 13:49:39 GMT):
can't we write the blocks into a temporary ledger?
jyellick (Mon, 09 Jul 2018 13:50:05 GMT):
I don't disagree that it's possible... only that I think this adds significantly to the complexity, and I'm not sure that the benefit is worth it.
yacovm (Mon, 09 Jul 2018 13:50:46 GMT):
i.e:
1) Follower starts with latest config block
2) Follower requests snapshot and participates in the cluster
3) Follower writes all the created blocks in the meantime to a temporary ledger
4) Follower in the background calls Deliver and then writes all the blocks sequentially into the real ledger
jeroiraz (Mon, 09 Jul 2018 13:51:14 GMT):
the ledger could still be continuos but just maintain the latest block in memory or other temporal storage
yacovm (Mon, 09 Jul 2018 13:51:32 GMT):
@jeroiraz he means the blocks that are written in the meantime
yacovm (Mon, 09 Jul 2018 13:52:15 GMT):
we don't actually even need a real temporary ledger though, just a.... file ;)
jyellick (Mon, 09 Jul 2018 13:54:32 GMT):
It's a valid scheme, it would work, but it's more complex, and I'm not sure that network operators would even like it. We used the model you describe above (or a similar one) in 0.5/0.6 for state transfer, and it was very confusing for users, and the failures were hard to deal with. My feeling is that network operators would prefer safety and simplicity over speed to recovery. If we find that the complete-sync before participating in consensus is a problem, we can always revisit. I'm not convinced that there is much lost work, as the sync before consenting behavior is basically a few lines of code.
jyellick (Mon, 09 Jul 2018 13:54:32 GMT):
It's a valid scheme, it would work, but it's more complex, and I'm not sure that network operators would even like it. We used the model you describe above (or a similar one) in 0.5/0.6 for state transfer, and it was very confusing for users, and the failures were hard to deal with. My feeling is that network operators would prefer safety and simplicity over speed to recovery. If we find that the complete-sync before participating in consensus is a problem, we can always revisit. I'm not convinced that there is much lost work, as the sync before consenting behavior is basically just a few lines of code.
jyellick (Mon, 09 Jul 2018 13:56:55 GMT):
In general, I think we try to be too clever, too soon. In my opinion, we need to deliver a bare minimum, simplest viable Raft implementation first, learn from it, and prioritize work from there.
yacovm (Mon, 09 Jul 2018 13:57:02 GMT):
ok.. understood. so another thing - in CFT we always always accept the block that is outputted from the `Ready` channel, but how is it going to work in BFT? All nodes should validate the block, in order to collect signatures...
jeroiraz (Mon, 09 Jul 2018 13:57:15 GMT):
indeed that requirement of first sync and then validate wouldn't be reflected into efficiency, given the consensus protocol is leader based and without actual agreement with the followers
yacovm (Mon, 09 Jul 2018 13:57:35 GMT):
@jeroiraz may you introduce yourself? :)
yacovm (Mon, 09 Jul 2018 13:57:43 GMT):
Never seen your around
yacovm (Mon, 09 Jul 2018 13:57:43 GMT):
Never seen you around
jyellick (Mon, 09 Jul 2018 13:58:46 GMT):
> but how is it going to work in BFT? All nodes should validate the block, in order to collect signatures...
Each replica should validate the transactions in the block before replying with a prepare. To address non-determinism for instance around validating timestamps, the replica should reply with a prepare once a weak cert of signatures is received, regardless of their belief in the block's transaction validity.
yacovm (Mon, 09 Jul 2018 13:59:20 GMT):
so the OSNs would intercept the communication messages and validate?
yacovm (Mon, 09 Jul 2018 13:59:20 GMT):
so the OSNs would intercept the communication messages before they get into the consensus instance, and validate?
jeroiraz (Mon, 09 Jul 2018 13:59:43 GMT):
sure, I'm Jeronimo Irazabal, I'm a software engineer working for IBM Argentina for around three years. I've been working with HF for some time, mostly with research projects e.g. extended the fabric to support SQL on chaincodes. I found the proposal to incorporate Raft very interesting and contacted Kostas to collaborate on it
jyellick (Mon, 09 Jul 2018 14:03:12 GMT):
I've been putting off heading into the office for too long need to drive in -- will be back online shortly
kostas (Mon, 09 Jul 2018 14:08:03 GMT):
Good morning. Let me try to catch up on what I've missed.
kostas (Mon, 09 Jul 2018 14:17:28 GMT):
> you can't do a concurrent change in the original way because you are supposed to only update 1 raft node at a time in small clusters, so in the original way
Ah, so put differently - unless you actually want to add or remove a node, work all those other modifications through a mechanism that doesn't trigger raft ConfChangeAddNode/ConfChangeRemoveNode messages in the Raft state machine. (And that mechanism for us is Fabric configuration messages.)
kostas (Mon, 09 Jul 2018 14:17:46 GMT):
Am I getting it right?
yacovm (Mon, 09 Jul 2018 14:20:45 GMT):
yes
yacovm (Mon, 09 Jul 2018 14:21:16 GMT):
another thing....
yacovm (Mon, 09 Jul 2018 14:22:01 GMT):
if we do a membership expansion we do it in 2 steps, right? first we authorize the node for fabric, and then we do a "propose" that proposes a config change to the raft cluster
yacovm (Mon, 09 Jul 2018 14:22:20 GMT):
when you said before that we can't have a txn slip in between i agreed with you, but we were both wrong IMO
yacovm (Mon, 09 Jul 2018 14:23:55 GMT):
> So, Fabric config change adds OSN 4 to the mix.
> But Raft doesn't see OSN 4 as part of its cluster yet.
but what if the new txn comes from OSN 1 which isn't the leader?
kostas (Mon, 09 Jul 2018 14:24:38 GMT):
The new transaction as in: a request to order an envelope?
yacovm (Mon, 09 Jul 2018 14:24:42 GMT):
yes
kostas (Mon, 09 Jul 2018 14:25:30 GMT):
I have a comment, but let me check the code real quick.
yacovm (Mon, 09 Jul 2018 14:26:24 GMT):
since we call `Step` asynchronously - we need a way to ensure that propose config is called immediately after which a config block is cut, somehow... no?
kostas (Mon, 09 Jul 2018 14:27:18 GMT):
One thing at a time.
yacovm (Mon, 09 Jul 2018 14:27:57 GMT):
or - maybe it works because we'll do an apply config change whenever we get the transaction bundle (config block) from the `Ready` ?
kostas (Mon, 09 Jul 2018 14:27:59 GMT):
> I have a comment, but let me check the code real quick.
So my comment here is that this node cannot successfully invoke its FSM to find out who the leader is, so how can he route its request to the leader?
yacovm (Mon, 09 Jul 2018 14:28:35 GMT):
no, forget the new node... I'm just saying - how do we guarantee we do an applyConfigChange right after the config block?
yacovm (Mon, 09 Jul 2018 14:29:14 GMT):
to do that we need to ProposeConfChange
yacovm (Mon, 09 Jul 2018 14:29:37 GMT):
but how do we make sure our ProposeConfChange gets right after the config block itself?
kostas (Mon, 09 Jul 2018 14:29:40 GMT):
The leader maintains three queues: one for Raft config messages, one for Fabric config messages, one for Fabric normal messages.
kostas (Mon, 09 Jul 2018 14:29:54 GMT):
These are listed in order of descending priority.
kostas (Mon, 09 Jul 2018 14:30:31 GMT):
So, when you have no Fabric config messages in, that means no Raft config messages.
kostas (Mon, 09 Jul 2018 14:31:27 GMT):
In this stage, normal Fabric messages reach the leader, and they attempt to cut blocks out of them and order them per the standard process.
kostas (Mon, 09 Jul 2018 14:31:43 GMT):
Now, when a Fabric configuration message reaches the leader, it gets put into that second queue.
yacovm (Mon, 09 Jul 2018 14:32:00 GMT):
aha... so the leader can then immedately enqueue a config propose for raft once it sees a fabric config, right?
kostas (Mon, 09 Jul 2018 14:32:11 GMT):
Which takes priority over the third one, and in fact causes us to stop processing/order —
kostas (Mon, 09 Jul 2018 14:32:14 GMT):
Correct.
yacovm (Mon, 09 Jul 2018 14:32:28 GMT):
is these 3 queues in the doc? :thinking_face:
yacovm (Mon, 09 Jul 2018 14:32:28 GMT):
is these 3 concept queues in the doc? :thinking_face:
kostas (Mon, 09 Jul 2018 14:32:33 GMT):
Yes.
kostas (Mon, 09 Jul 2018 14:32:58 GMT):
Footnote 16, page 30. All the good stuff is in the footnotes, in typical DFW fashion.
kostas (Mon, 09 Jul 2018 14:34:14 GMT):
Let me go back to the things I missed earlier, but if you have any more questions, please post them here. I truly appreciate the double-checking and the follow-ups, helps us make sure we're not missing anything.
yacovm (Mon, 09 Jul 2018 14:34:19 GMT):
wait wait
yacovm (Mon, 09 Jul 2018 14:34:21 GMT):
before that
yacovm (Mon, 09 Jul 2018 14:35:31 GMT):
so if the leader sees a fabric config, and then before it has a chance to propagate the propose config change - it dies - the new leader should do that afterwards when it establishes command.
it needs to look at the head of the log and then do the propose config change itself, right?
kostas (Mon, 09 Jul 2018 14:36:26 GMT):
Correct.
yacovm (Mon, 09 Jul 2018 14:37:55 GMT):
and what do you think about that new follower that is added which Jason commented against?
yacovm (Mon, 09 Jul 2018 14:38:17 GMT):
the one that Jeronimo also chimed into the discussion ;)
kostas (Mon, 09 Jul 2018 14:38:35 GMT):
Ah, link?
yacovm (Mon, 09 Jul 2018 14:38:48 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=wCmwSAYZCjLvie32J
yacovm (Mon, 09 Jul 2018 14:39:08 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=bnr6acnAkpHcCxyZd
kostas (Mon, 09 Jul 2018 14:39:18 GMT):
Oh, this is from earlier in the discussion here, right?
yacovm (Mon, 09 Jul 2018 14:39:23 GMT):
yes
kostas (Mon, 09 Jul 2018 14:39:27 GMT):
I'm working through those, gimme a sec.
kostas (Mon, 09 Jul 2018 14:41:30 GMT):
> but i think the raft plumbing isn't elaborated enough IMO
That criticism is accurate. I literally had to draw all the interactions going on in the `raftexample` in paper to figure things out, and that was a multi-day process with a lot of digging around. I cannot think of a good way to capture all of this into the document, without turning it into a 70-pg opus, and even then, I'm not sure it'll be as effective as looking at this example. Ultimately, I think that studying the `raftexample` is a rite of passage we all have to go through (which is why I'm suggesting it in every opportunity). If questions on the Raft's internals are still there after studying, we're here to sort them out.
yacovm (Mon, 09 Jul 2018 14:42:23 GMT):
that's what I did today :)
kostas (Mon, 09 Jul 2018 14:42:36 GMT):
Ah, excellent.
kostas (Mon, 09 Jul 2018 14:42:48 GMT):
It took me more days than that unfortunately.
yacovm (Mon, 09 Jul 2018 14:43:08 GMT):
I never said I understand it all....
yacovm (Mon, 09 Jul 2018 14:43:21 GMT):
anyway , about my question with the follower joining late?
kostas (Mon, 09 Jul 2018 14:43:29 GMT):
(Getting there.)
kostas (Mon, 09 Jul 2018 14:44:03 GMT):
> also 1 more thing I'm missing - all OSNs need to have the entire ledger since they serve Deliver requests. Obviously we can't use a snapshot for that because the data is huge so the OSN that joins would need to call Deliver on its own, right? But at the same time - it may serve Deliver() requests from peers for blocks it doesn't have. I guess we would need to return `SERVICE_UNAVAILABLE` ?
kostas (Mon, 09 Jul 2018 14:44:22 GMT):
(Keep in mind that I haven't scrolled through the rest of the convo, so I'm just addressing things one-by-one.)
kostas (Mon, 09 Jul 2018 14:44:49 GMT):
So for this one, this is where the snapshots will come in.
kostas (Mon, 09 Jul 2018 14:46:02 GMT):
And snapshots here, the way we've defined them, are basically checkpoints that allow the node that's catching up to issue an educated Deliver request.
yacovm (Mon, 09 Jul 2018 14:47:27 GMT):
yeah i get that, but what i was asking is something else - why do we have to first attempt to catch up to the snapshot via deliver and only *then* request a snapshot , instead of just request a snapshot immediately at startup, and in the background - sync the ledger ?
kostas (Mon, 09 Jul 2018 14:52:36 GMT):
I'm not sure I agree with the phrase "attempt to catch up to the snapshot via deliver and then request a snapshot."
kostas (Mon, 09 Jul 2018 14:52:43 GMT):
Let me take another stab at what's going on.
kostas (Mon, 09 Jul 2018 14:52:48 GMT):
Every Raft node keeps a number of X most recent entries in its Raft storage (and you garbage collect these every now and then, so as not to keep them all in memory, at least in the default, provided implementation).
kostas (Mon, 09 Jul 2018 14:52:52 GMT):
If the leader finds out your log ends at Raft entry 1 and we're now at Raft entry 200, and they only keep the last 20 Raft entries in memory, the leader will go "ah I need to send this guy my most recent snapshot".
kostas (Mon, 09 Jul 2018 14:53:15 GMT):
So there is a good chance that when you first join a very active network, then the very first you'll be sent is a snapshot.
kostas (Mon, 09 Jul 2018 14:53:44 GMT):
And you're expected to act on it, by invoking Deliver based on what that snapshot says (which is basically "we're up to block 210938, please sync up").
kostas (Mon, 09 Jul 2018 14:55:17 GMT):
Now, going to this:
kostas (Mon, 09 Jul 2018 14:55:18 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=bnr6acnAkpHcCxyZd
kostas (Mon, 09 Jul 2018 14:57:35 GMT):
(Looking at my notes for something.)
kostas (Mon, 09 Jul 2018 15:03:31 GMT):
> but I think it's sub-optimal because while we sync the blocks in the ledger, the node doesn't sync with the raft cluster, right?
At the risk of sounding like an idiot, why is this sub-optimal?
kostas (Mon, 09 Jul 2018 15:03:43 GMT):
I want to make sure I'm not missing the goal here.
kostas (Mon, 09 Jul 2018 15:04:24 GMT):
When you sync blocks in the ledger, this is actually you trying to sync up with the Raft cluster.
yacovm (Mon, 09 Jul 2018 15:06:42 GMT):
that's not what i understood that happens
yacovm (Mon, 09 Jul 2018 15:08:11 GMT):
I think I mis-understood and thought we don't request the snapshot right away
yacovm (Mon, 09 Jul 2018 15:08:16 GMT):
or something like that
kostas (Mon, 09 Jul 2018 15:08:54 GMT):
(Understood. Feel free to ask more questions around that if there's anything in particular you'd like me to clarify. In the meantime, I'm looking at the rest of your messages from earlier today, and will keep posting.)
kostas (Mon, 09 Jul 2018 15:10:07 GMT):
> I considered for quite a while whether we should require that the node complete syncing before participating in the cluster, as it's not strictly necessary that the sync complete, so long as the snapshot contains the latest config block, and a hash of the previous block header. However, this would require even more modifications to the ledger, and the failure cases become much more complex. Until we have a more compelling reason to do so, I think halting the follower's participation in consensus until it has completely caught up is the best solution.
kostas (Mon, 09 Jul 2018 15:12:51 GMT):
I see that @jyellick has covered all of that, excellent. I could sense that this is where you where getting at -- active participation of a lagging node in ordering even before the catching up is complete. In a footnote somewhere in the snapshotting part of the doc, I have a reference as to what the Snapshot message might be extended too eventually -- that's all a product of a similar discussion I had with Jason when we were trying to figure out how to make active ordering participation work.
kostas (Mon, 09 Jul 2018 15:15:13 GMT):
> In general, I think we try to be too clever, too soon. In my opinion, we need to deliver a bare minimum, simplest viable Raft implementation first, learn from it, and prioritize work from there.
This is a perfect summary of the approach we should be adopting. We have a lot to juggle with as is, which is why you also see me shooting down a few suggestions in the doc comments.
kostas (Mon, 09 Jul 2018 15:15:46 GMT):
I get there are things we can improve on, but unless the current suggestion is broken, it'd be wise to defer on those improvements.
kostas (Mon, 09 Jul 2018 15:17:22 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=R3EedLhrbPacMAuq2
kostas (Mon, 09 Jul 2018 15:17:27 GMT):
Not sure I got that?
kostas (Mon, 09 Jul 2018 15:17:41 GMT):
@jeroiraz Hi! Glad you joined.
kostas (Mon, 09 Jul 2018 15:18:58 GMT):
> Not sure I got that?
(Not the first sentence, the rest.)
jyellick (Mon, 09 Jul 2018 15:19:00 GMT):
> Not sure I got that?
Certainly BFT replicas need to validate that the block content is legitimate. That the transactions are valid, and the leader is not simply injecting junk to give the illusion of progress. However, despite our best intentions, we cannot guarantee that validation is deterministic. Obviously we try our very best, but, certain checks, like those against timestamp will necessarily not be deterministic (unless we take significant steps to incorporate a BFT clock).
jyellick (Mon, 09 Jul 2018 15:19:00 GMT):
> Not sure I got that?
Certainly BFT replicas need to validate that the block content is legitimate.
> (Not the first sentence, the rest.) [not deleting, but this is the rest]
That the transactions are valid, and the leader is not simply injecting junk to give the illusion of progress. However, despite our best intentions, we cannot guarantee that validation is deterministic. Obviously we try our very best, but, certain checks, like those against timestamp will necessarily not be deterministic (unless we take significant steps to incorporate a BFT clock).
jyellick (Mon, 09 Jul 2018 15:19:00 GMT):
> Not sure I got that?
Certainly BFT replicas need to validate that the block content is legitimate. That the transactions are valid, and the leader is not simply injecting junk to give the illusion of progress.
> (Not the first sentence, the rest.) [not deleting, but this is the rest]
However, despite our best intentions, we cannot guarantee that validation is deterministic. Obviously we try our very best, but, certain checks, like those against timestamp will necessarily not be deterministic (unless we take significant steps to incorporate a BFT clock).
jyellick (Mon, 09 Jul 2018 15:19:00 GMT):
> Not sure I got that?
Certainly BFT replicas need to validate that the block content is legitimate. That the transactions are valid, and the leader is not simply injecting junk to give the illusion of progress. But rather than vote for a view change on perceived junk, the replica should probably simply not prepare, and allow the view change timer to handle view change.
> (Not the first sentence, the rest.) [not deleting, but this is the rest]
However, despite our best intentions, we cannot guarantee that validation is deterministic. Obviously we try our very best, but, certain checks, like those against timestamp will necessarily not be deterministic (unless we take significant steps to incorporate a BFT clock).
kostas (Mon, 09 Jul 2018 15:20:31 GMT):
Ah, got it.
yacovm (Mon, 09 Jul 2018 15:21:51 GMT):
so.... the reason i asked about the BFT thing - is that I think that we it makes sense to build a communication infrastructure that can be reused as much as possible for BFT later on
yacovm (Mon, 09 Jul 2018 15:21:58 GMT):
or for any kind of block-ordering OSN
kostas (Mon, 09 Jul 2018 15:26:53 GMT):
Understood. I don't see how this particular bit can be retrofitted for BFT in a way that's meaningful, but if you were intent to do that, I presume you would add logic to the server's Step RPC method. It would check whether the incoming blocks are valid and have enough signatures. If they're not valid, it would reject them and prevent them from entering the FSM. If they were valid but not carrying enough signatures, it would add its own signature, and `Step` them on other nodes or other nodes, until it got them back with enough signatures, at which point it would allow them to enter its FSM.
yacovm (Mon, 09 Jul 2018 15:27:39 GMT):
I was thinking of having a interception layers for the RPCs in general
yacovm (Mon, 09 Jul 2018 15:27:51 GMT):
that's the cleanest way to handle the mapping of the IDs and TLS certs, etc.
kostas (Mon, 09 Jul 2018 15:29:39 GMT):
Perhaps naively, I think this part should be straightforward when BFT comes. What matters now is that we establish a flow that allows the OSNs to do OSN-OSN comms via RPCs, and then we can adjust these RPCs to do what we wish.
kostas (Mon, 09 Jul 2018 15:30:19 GMT):
> I was thinking of having a interception layers for the RPCs in general
Yes, this is something that has crossed my mind as well. I've played around with interceptors when I read this one last year: https://about.sourcegraph.com/go/grpc-in-production-alan-shreve/
yacovm (Mon, 09 Jul 2018 15:30:27 GMT):
lol
yacovm (Mon, 09 Jul 2018 15:30:45 GMT):
`Why do REST APIs suck?`
yacovm (Mon, 09 Jul 2018 15:31:08 GMT):
you should totally publish that in #composer
kostas (Mon, 09 Jul 2018 15:31:20 GMT):
Well, he's transcribing a presentation and non-opinionated presentations are a bit boring ;)
yacovm (Mon, 09 Jul 2018 15:32:04 GMT):
we have an interception layer in fabric even now.... the chaincode access control, wraps the infamous `ChaincodeSupport` object ;)
yacovm (Mon, 09 Jul 2018 15:32:04 GMT):
we have an interception layer in fabric even now.... the chaincode access control, wraps the infamous `ChaincodeSupport` object ;)
https://github.com/hyperledger/fabric/blob/release-1.2/peer/node/start.go#L627-L630
yacovm (Mon, 09 Jul 2018 15:32:15 GMT):
and checks TLS pinning
kostas (Mon, 09 Jul 2018 15:32:24 GMT):
Ah, I didn't know that.
kostas (Mon, 09 Jul 2018 15:33:06 GMT):
Anyway, I only implemented them for logging in my own tests and I didn't see them adding too much complexity. If we decide we wish to go forward with that, and we can JIRA this properly so that we know what we're doing, then I'm for it.
yacovm (Mon, 09 Jul 2018 15:33:30 GMT):
that's also how the authentication handlers for endorser work btw- we have a wrapper pipeling
yacovm (Mon, 09 Jul 2018 15:33:30 GMT):
that's also how the authentication handlers for endorser work btw- we have a wrapper pipeline
kostas (Mon, 09 Jul 2018 15:34:38 GMT):
Add that to the already big list of things I wasn't aware of.
kostas (Mon, 09 Jul 2018 15:41:04 GMT):
As a status check, would we say we need couple of more days till we kickstart this, or are we more or less good to go? @yacovm @C0rWin @guoger @jeroiraz
yacovm (Mon, 09 Jul 2018 15:41:56 GMT):
wait - what about that node ID change
yacovm (Mon, 09 Jul 2018 15:42:31 GMT):
that we don't do a raft ProposeConfig when we change an existing node's certificate/endpoint
yacovm (Mon, 09 Jul 2018 15:42:35 GMT):
we only do a fabric config
yacovm (Mon, 09 Jul 2018 15:42:41 GMT):
are you good with it?
kostas (Mon, 09 Jul 2018 15:42:54 GMT):
Ah, yes. I'm good.
yacovm (Mon, 09 Jul 2018 15:42:58 GMT):
also I don't understand what is the bottom line about the net.Listener
yacovm (Mon, 09 Jul 2018 15:43:02 GMT):
do we enforce TLS?
yacovm (Mon, 09 Jul 2018 15:43:05 GMT):
or allow diff. listener?
yacovm (Mon, 09 Jul 2018 15:43:11 GMT):
your comment was cryptic :)
kostas (Mon, 09 Jul 2018 15:43:39 GMT):
You remind me that I haven't addressed the comments in the doc since last night - will do that now.
kostas (Mon, 09 Jul 2018 15:44:39 GMT):
Discussed this with Gari and he's also voting for going all in on enforcing TLS, and using the same listener.
yacovm (Mon, 09 Jul 2018 15:45:23 GMT):
OK
kostas (Tue, 10 Jul 2018 10:12:04 GMT):
Sent an invite for a 30-min Zoom session tomorrow at 8am EDT to kick things off and discuss any outstanding questions: https://zoom.us/j/7432937602
yacovm (Tue, 10 Jul 2018 10:16:13 GMT):
@kostas tomorrow is a very bad day for me for meetings :(
yacovm (Tue, 10 Jul 2018 10:16:17 GMT):
can't we do it today?
kostas (Tue, 10 Jul 2018 10:16:55 GMT):
Sure. Depends on whether @guoger and @C0rWin are available.
yacovm (Tue, 10 Jul 2018 10:17:08 GMT):
@C0rWin is on a trip abroad all week
kostas (Tue, 10 Jul 2018 10:17:26 GMT):
Ah. Perhaps it's best to defer till this coming Monday then.
kostas (Tue, 10 Jul 2018 10:17:26 GMT):
Ah. Perhaps it's best to defer till this coming Monday then?
yacovm (Tue, 10 Jul 2018 10:17:33 GMT):
that's fine with me
guoger (Tue, 10 Jul 2018 10:17:45 GMT):
All those options work for me
kostas (Tue, 10 Jul 2018 10:18:04 GMT):
You know what, since you're both here, let's have a quick meeting today.
kostas (Tue, 10 Jul 2018 10:18:23 GMT):
And Yacov can bring Artem on board?
kostas (Tue, 10 Jul 2018 10:18:28 GMT):
(And we'll meet again.)
yacovm (Tue, 10 Jul 2018 10:18:52 GMT):
I suggest we not distribute work this week though since Artem is out ;)
yacovm (Tue, 10 Jul 2018 10:18:59 GMT):
but only talk technical stuff
guoger (Tue, 10 Jul 2018 10:19:05 GMT):
Sure, If we do it today, Can we do it a bit later, 9?
yacovm (Tue, 10 Jul 2018 10:19:34 GMT):
How come you're so early @kostas ?
kostas (Tue, 10 Jul 2018 10:19:54 GMT):
I was about to say, 9 is a bit tricky because I've been up all night :upside_down:
kostas (Tue, 10 Jul 2018 10:19:54 GMT):
(I was about to say, 9 is a bit tricky because I've been up all night.)
guoger (Tue, 10 Jul 2018 10:19:57 GMT):
His routine is always a myth to me
yacovm (Tue, 10 Jul 2018 10:20:10 GMT):
oh... you still haven't gone asleep?
kostas (Tue, 10 Jul 2018 10:20:42 GMT):
Yeah, my sleep schedule is all messed up - something I want to fix during this break.
kostas (Tue, 10 Jul 2018 10:20:42 GMT):
Yeah, my sleep schedule is slightly messed up - something I want to fix during this break.
guoger (Tue, 10 Jul 2018 10:20:52 GMT):
Ok, let’s do 8
kostas (Tue, 10 Jul 2018 10:21:22 GMT):
Alright, same link at 8.
C0rWin (Tue, 10 Jul 2018 10:21:41 GMT):
Hey, thanks f needed I can try to attend
yacovm (Tue, 10 Jul 2018 10:21:50 GMT):
so 2 hours from now?
C0rWin (Tue, 10 Jul 2018 10:22:10 GMT):
I think will
kostas (Tue, 10 Jul 2018 10:22:10 GMT):
Artem, no rush - if you're on a trip, this can wait.
guoger (Tue, 10 Jul 2018 10:22:18 GMT):
1hr 40min I suppose
kostas (Tue, 10 Jul 2018 10:22:24 GMT):
But up to you. We're meeting in 1h40m, yes.
C0rWin (Tue, 10 Jul 2018 10:22:44 GMT):
I have meetings with GBS guys all day long
C0rWin (Tue, 10 Jul 2018 10:23:14 GMT):
But it will be great to have some real staff
yacovm (Tue, 10 Jul 2018 10:23:15 GMT):
sounds fun ;)
C0rWin (Tue, 10 Jul 2018 10:23:31 GMT):
Disgusting
C0rWin (Tue, 10 Jul 2018 10:24:46 GMT):
Hangout?
yacovm (Tue, 10 Jul 2018 10:24:59 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=zbSqNgPGNqPLDpr9F
kostas (Tue, 10 Jul 2018 10:25:05 GMT):
I believe the cool kids are now referring to it as Zoom: https://zoom.us/j/7432937602
kostas (Tue, 10 Jul 2018 10:26:08 GMT):
Copying @jeroiraz as well.
kostas (Tue, 10 Jul 2018 10:26:08 GMT):
Copying @jeroiraz as well. (But no worries if you can't make it, we shall repeat this.)
yacovm (Tue, 10 Jul 2018 10:32:27 GMT):
Links for the spreadsheet, @kostas ? :)
kostas (Tue, 10 Jul 2018 10:33:08 GMT):
https://docs.google.com/spreadsheets/d/1R02-xjdl4hNw90kN4NZ1kX0f4QoIYn51_1UHlwy-GQs/edit?usp=sharing
kostas (Tue, 10 Jul 2018 10:33:08 GMT):
https://docs.google.com/spreadsheets/d/1R02-xjdl4hNw90kN4NZ1kX0f4QoIYn51_1UHlwy-GQs/edit?usp=sharing
kostas (Tue, 10 Jul 2018 10:33:08 GMT):
Proposed JIRA breakdown for Raft: https://docs.google.com/spreadsheets/d/1R02-xjdl4hNw90kN4NZ1kX0f4QoIYn51_1UHlwy-GQs/edit?usp=sharing
kostas (Tue, 10 Jul 2018 10:33:08 GMT):
Proposed JIRA breakdown for Raft: https://docs.google.com/spreadsheets/d/1R02-xjdl4hNw90kN4NZ1kX0f4QoIYn51_1UHlwy-GQs/edit?usp=sharing
kostas (Tue, 10 Jul 2018 10:33:08 GMT):
Proposed JIRA breakdown for Raft: https://docs.google.com/spreadsheets/d/1R02-xjdl4hNw90kN4NZ1kX0f4QoIYn51_1UHlwy-GQs/edit?usp=sharing
kostas (Tue, 10 Jul 2018 10:33:29 GMT):
(You have write access as well.)
guoger (Tue, 10 Jul 2018 11:40:29 GMT):
Sorry, I think I’ll be 5 min late
kostas (Tue, 10 Jul 2018 11:40:57 GMT):
No problem.
guoger (Tue, 10 Jul 2018 12:05:48 GMT):
connecting
C0rWin (Tue, 10 Jul 2018 12:41:19 GMT):
guys, I won't be able to reconnect as I have to move to the next meeting
C0rWin (Tue, 10 Jul 2018 12:41:22 GMT):
will catch up later
kostas (Tue, 10 Jul 2018 12:41:23 GMT):
Will bring up the meeting again, we got disconnected.
kostas (Tue, 10 Jul 2018 12:41:36 GMT):
Gimme 1'.
kostas (Tue, 10 Jul 2018 12:49:41 GMT):
(I think I've accidentally stopped the meeting for all, you should be able to rejoin at the same link though.)
kostas (Tue, 10 Jul 2018 13:09:41 GMT):
Rough call summary (correct/expand with anything I've missed): Instead of the top-down approach proposed in the spreadsheet, perhaps a bottom-up approach might make more sense, as it will be less intrusive on Fabric core if we don't make it in time for the 1.3 cut. My concern is that we're missing the ability to drive this with integration tests from the get go, but perhaps this concern is exaggerated. Yacov and Jay to take a stab at how the first couple of weeks in a bottom-up approach will look like, and I'll review with Yacov later today/tomorrow. If anyone else has comments on the JIRA breakdown, post here or directly on the spreadsheet.
kostas (Tue, 10 Jul 2018 13:10:39 GMT):
kostas (Tue, 10 Jul 2018 13:11:30 GMT):
kostas (Tue, 10 Jul 2018 13:14:06 GMT):
@guoger: This issue is related to your first question: https://github.com/coreos/etcd/issues/9809
kostas (Tue, 10 Jul 2018 13:14:06 GMT):
@guoger: This issue is related to your first question from the call: https://github.com/coreos/etcd/issues/9809
NoLimitHoldem (Wed, 11 Jul 2018 06:16:45 GMT):
Has joined the channel.
jayeshjawale95 (Wed, 11 Jul 2018 07:32:35 GMT):
Has joined the channel.
kostas (Wed, 11 Jul 2018 15:26:24 GMT):
> Yacov and Jay to take a stab at how the first couple of weeks in a bottom-up approach will look like, and I'll review with Yacov later today/tomorrow
An update: Yacov and I discussed the new proposed plan. Let us proceed as described in the sheet "Week 1'-2'" on the spreadsheet. We have a difference of opinion on line 12 and the usefulness of extended unit tests - we'll be reaching out to Matt to get his take on it. JIRA stories for week 1 items to come tomorrow, and work on this to begin this Monday.
yuki-kon (Wed, 11 Jul 2018 16:18:50 GMT):
Has joined the channel.
muralisr (Wed, 11 Jul 2018 22:40:28 GMT):
Has joined the channel.
huikang (Thu, 12 Jul 2018 02:09:35 GMT):
Has joined the channel.
WadeLu (Thu, 12 Jul 2018 07:33:03 GMT):
Has joined the channel.
mnarayan (Thu, 12 Jul 2018 09:00:36 GMT):
Has joined the channel.
Senthil1 (Thu, 12 Jul 2018 17:32:43 GMT):
Has joined the channel.
sergefdrv (Fri, 13 Jul 2018 12:41:59 GMT):
Has joined the channel.
yacovm (Sun, 15 Jul 2018 11:43:03 GMT):
@guoger @kostas @C0rWin I opened JIRAs for the 1st week:
https://jira.hyperledger.org/browse/FAB-11163
https://jira.hyperledger.org/browse/FAB-11162
https://jira.hyperledger.org/browse/FAB-11161
https://jira.hyperledger.org/browse/FAB-11160
https://jira.hyperledger.org/browse/FAB-11159
yacovm (Sun, 15 Jul 2018 11:43:38 GMT):
I think we can open new JIRAs that consolidate the various components based on our progress
yacovm (Sun, 15 Jul 2018 11:44:11 GMT):
Also I think the membership and communication modules should be implemented by the same person since they should be somewhat tightly coupled
yacovm (Mon, 16 Jul 2018 06:30:06 GMT):
@guoger do you want to split the work?
guoger (Mon, 16 Jul 2018 06:37:32 GMT):
Sure, it looks like 11163 and 11162 should also be implemented by the same person, I could take this part
yacovm (Mon, 16 Jul 2018 06:40:47 GMT):
lol
yacovm (Mon, 16 Jul 2018 06:40:56 GMT):
I don't think so.... on the contrary
yacovm (Mon, 16 Jul 2018 06:41:19 GMT):
I'd say that the consenter and the FSM are very well separated
yacovm (Mon, 16 Jul 2018 06:41:43 GMT):
also - FAB-11159 is just an interface declaration, it's not really much work there
yacovm (Mon, 16 Jul 2018 06:41:52 GMT):
You expressed interest to do the FSM, no?
yacovm (Mon, 16 Jul 2018 06:42:19 GMT):
why don't you take the FSM, @C0rWin will take the consenter FAB-11163 and I'll do 60 and 61?
yacovm (Mon, 16 Jul 2018 06:42:56 GMT):
of course - you can break down FAB-11162 to sub-tasks as you see fit
yacovm (Mon, 16 Jul 2018 06:43:15 GMT):
maybe I should turn 11159 to a sub-task of 11160 :thinking:
guoger (Mon, 16 Jul 2018 06:44:10 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=TAAu9rtnKMLjLSo96) @yacovm if FSM interfaces are well defined...
guoger (Mon, 16 Jul 2018 06:44:25 GMT):
shouldn't them all be included to an epic?
yacovm (Mon, 16 Jul 2018 06:44:32 GMT):
the interfaces?
guoger (Mon, 16 Jul 2018 06:45:09 GMT):
nvm, i misunderstood `consenter` as `chain`
yacovm (Mon, 16 Jul 2018 06:45:19 GMT):
that's the chain actually
yacovm (Mon, 16 Jul 2018 06:45:45 GMT):
in kafka IIRC the chain and consenter are sort of together no?
guoger (Mon, 16 Jul 2018 06:46:13 GMT):
consenter is kinda singleton, which spawns chains
yacovm (Mon, 16 Jul 2018 06:46:24 GMT):
I meant the chain then
yacovm (Mon, 16 Jul 2018 06:46:24 GMT):
I meant the chain then https://github.com/hyperledger/fabric/blob/release-1.2/orderer/consensus/kafka/chain.go
The consenter is a thin wrapper https://github.com/hyperledger/fabric/blob/release-1.2/orderer/consensus/kafka/consenter.go
guoger (Mon, 16 Jul 2018 06:46:29 GMT):
and I would imagine `chain` to hold `fsm` instance
guoger (Mon, 16 Jul 2018 06:46:57 GMT):
and relay messages to communication layer
yacovm (Mon, 16 Jul 2018 06:46:58 GMT):
the spawning code is pretty tiny though... it just creates a new chain and returns it
yacovm (Mon, 16 Jul 2018 06:47:01 GMT):
no....
yacovm (Mon, 16 Jul 2018 06:47:15 GMT):
why does the chain need to relay messages to the communication layer?
yacovm (Mon, 16 Jul 2018 06:47:32 GMT):
we relay everything to the FSM don't we?
yacovm (Mon, 16 Jul 2018 06:48:13 GMT):
it will find out who is the leader and then relay to it the envelope
yacovm (Mon, 16 Jul 2018 06:48:35 GMT):
I think we shouldn't make the chain talk to the `raft.Node`
yacovm (Mon, 16 Jul 2018 06:48:55 GMT):
and if we can just put that logic in the FSM - that's easier code wise don't you think?
guoger (Mon, 16 Jul 2018 06:52:56 GMT):
w.r.t communication, raft service is registered to our gRPC server. whenever it receives a message, it inspect the header and dispatch it to underlying `chain`. And `chain` would decide whether sends it to `fsm`, or sends it to the follower
guoger (Mon, 16 Jul 2018 06:54:17 GMT):
I'm not sure how would you handle dispatching if fsm has direct access to communication?
yacovm (Mon, 16 Jul 2018 06:54:56 GMT):
you already said the reason why I want the FSM to do the decision and not the chain ;)
yacovm (Mon, 16 Jul 2018 06:55:06 GMT):
> whenever it receives a message, it inspect the header and dispatch it to underlying `chain`. And `chain` would decide whether sends it to `fsm`, or sends it to the follower
yacovm (Mon, 16 Jul 2018 06:55:48 GMT):
you see - if you *anyway* forward the message to the FSM - then the FSM can decide whether to propose it, or to send it to another node, and it can do that since it has access to the raft communication layer
yacovm (Mon, 16 Jul 2018 06:55:54 GMT):
that makes the chain logic simpler
yacovm (Mon, 16 Jul 2018 06:56:02 GMT):
and the interfaces between them smaller
guoger (Mon, 16 Jul 2018 06:57:15 GMT):
the question is, to which `fsm` instance are you sending an ingress msg to?
yacovm (Mon, 16 Jul 2018 06:57:36 GMT):
I'm talking about the FSM wrapper module from FAB-11162
yacovm (Mon, 16 Jul 2018 06:58:06 GMT):
and @kostas said there is a way to extract from the `raft.Node` the current leader
yacovm (Mon, 16 Jul 2018 06:58:21 GMT):
so that makes sense the FSM do it....
yacovm (Mon, 16 Jul 2018 06:58:24 GMT):
no?
guoger (Mon, 16 Jul 2018 06:58:59 GMT):
hmm, are you saying we have a single wrapper that takes care of all channels?
yacovm (Mon, 16 Jul 2018 06:59:15 GMT):
no
yacovm (Mon, 16 Jul 2018 06:59:28 GMT):
I'm talking all the time in the context of a given channel
yacovm (Mon, 16 Jul 2018 06:59:42 GMT):
{ chain { FSM { raft.Node } } }
guoger (Mon, 16 Jul 2018 07:00:14 GMT):
do you wanna have a quick call?
yacovm (Mon, 16 Jul 2018 07:00:24 GMT):
when @C0rWin comes to the office
guoger (Mon, 16 Jul 2018 07:00:38 GMT):
sure
C0rWin (Mon, 16 Jul 2018 07:04:34 GMT):
In one hour?
guoger (Mon, 16 Jul 2018 07:05:04 GMT):
np :)
guoger (Mon, 16 Jul 2018 07:08:37 GMT):
@yacovm IIUC, each `fsm` would register to communication layer, and that's where you decide which `fsm` to dispatch a msg to?
yacovm (Mon, 16 Jul 2018 07:10:09 GMT):
yeah
yacovm (Mon, 16 Jul 2018 07:12:09 GMT):
though we can do something else frankly
yacovm (Mon, 16 Jul 2018 07:12:17 GMT):
we can have a comm layer per channel
yacovm (Mon, 16 Jul 2018 07:12:44 GMT):
that would be easier to implement IMO
yacovm (Mon, 16 Jul 2018 07:13:34 GMT):
i.e when you get a message you inspect the channel, and route it to the correct comm "instance" for that channel
guoger (Mon, 16 Jul 2018 07:17:09 GMT):
so I was thinking that we should do bookkeeping in the same place (`Registrar.chains` in this case)
guoger (Mon, 16 Jul 2018 07:18:17 GMT):
so we could avoid running into weird state due to async
guoger (Mon, 16 Jul 2018 07:19:00 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=FkQEcCJ9WnoYYAaXJ) @yacovm it depends on how you represent a `fsm` in comm layer, but it should be agnostic to comm users, no?
yacovm (Mon, 16 Jul 2018 07:19:28 GMT):
lets defer these questions to the call ;)
C0rWin (Mon, 16 Jul 2018 08:06:46 GMT):
@yacovm @guoger hangout?
yacovm (Mon, 16 Jul 2018 08:07:38 GMT):
sure
yacovm (Mon, 16 Jul 2018 08:08:18 GMT):
either that or telepathy
C0rWin (Mon, 16 Jul 2018 08:09:39 GMT):
I have another call meanwhile
guoger (Mon, 16 Jul 2018 08:14:40 GMT):
looking for a meeting room, just s sec
guoger (Mon, 16 Jul 2018 08:17:23 GMT):
let's zoom. hangout is blocked in China... vpn works slow today
guoger (Mon, 16 Jul 2018 08:17:43 GMT):
https://zoom.us/j/858466340
guoger (Mon, 16 Jul 2018 08:18:25 GMT):
oh sorry, just saw that @C0rWin has another meeting.
sergefdrv (Mon, 16 Jul 2018 08:19:38 GMT):
Hi all! We just published an initial version of implementation of a consensus protocol called MinBFT. Please have a look if you are interested: https://github.com/nec-blockchain/minbft. Any feedback is highly appreciated!
C0rWin (Mon, 16 Jul 2018 08:33:32 GMT):
@guoger @yacovm sorry guys, can we postpone for a while?
yacovm (Mon, 16 Jul 2018 08:36:11 GMT):
say when you can talk....
yacovm (Mon, 16 Jul 2018 08:36:14 GMT):
I'll adjust
guoger (Mon, 16 Jul 2018 08:38:22 GMT):
option 1: 12:00 - 13:00
option 2: 15:00 - 16:00
guoger (Mon, 16 Jul 2018 08:38:22 GMT):
option 1: 12:00 - 13:00
option 2: anytime between 15:00 - 18:00
guoger (Mon, 16 Jul 2018 08:38:25 GMT):
your time
guoger (Mon, 16 Jul 2018 08:38:33 GMT):
which works better for you?
guoger (Mon, 16 Jul 2018 08:39:48 GMT):
^ @yacovm @C0rWin
yacovm (Mon, 16 Jul 2018 09:01:47 GMT):
15:00 - 18:00
yacovm (Mon, 16 Jul 2018 09:46:24 GMT):
@guoger but regardless of details - do you agree to the fundamental work split that I proposed? (you do the FSM, Artem does the chain/consenter, and I do the comm/membership) ?
yacovm (Mon, 16 Jul 2018 09:46:36 GMT):
That is unrelated to how and where we plug things
guoger (Mon, 16 Jul 2018 09:55:31 GMT):
agreed :)
yacovm (Mon, 16 Jul 2018 09:58:12 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=aYmGxdMCnjALceMag) :thumbsup:
guoger (Mon, 16 Jul 2018 10:06:55 GMT):
I would imagine `RaftNode` ( `fsm` ) interfaces to be something like this:
```
Start()
Propose(block *cb.Block) error
Configure(...) error
Shutdown()
GetLeader() uint
```
and chain constructs `RaftNode` with `NewRaftNode(config raft.Config, commitC chan *cb.Block) RaftNode`, where `commitC` is used to receive output block from `RaftNode`.
So chain works roughly like this:
```
if leader = node.GetLeader(); leader == myID {
order()
} else {
// call comm layer to `Propose` tx to leader
}
```
guoger (Mon, 16 Jul 2018 10:06:55 GMT):
I would imagine `RaftNode` ( `fsm` ) interfaces to be something like this:
```
Start()
Propose(block *cb.Block) error
Configure(...) error
Shutdown()
GetLeader() uint
```
and chain constructs `RaftNode` with `NewRaftNode(config raft.Config, commitC chan *cb.Block) RaftNode`, where `commitC` is used to receive output block from `RaftNode`.
So chain works roughly like this:
```
if leader = node.GetLeader(); leader == myID {
order()
} else {
// call communication layer to `Propose` tx to leader
}
```
yacovm (Mon, 16 Jul 2018 10:30:07 GMT):
That's exactly my point... why do we need this `GetLeader` if we can just forward to the FSM anyway and let it do the routing?
yacovm (Mon, 16 Jul 2018 10:31:21 GMT):
and I thought the `chain` would hold an FSM? why does it need to hold the raft node?
guoger (Mon, 16 Jul 2018 10:31:53 GMT):
Raftnode is simply an alias of FSM
guoger (Mon, 16 Jul 2018 10:32:02 GMT):
not sure fsm is a good name...
yacovm (Mon, 16 Jul 2018 10:32:23 GMT):
it might not be but - since we have `raft.Node` in the code base... ;)
guoger (Mon, 16 Jul 2018 10:32:30 GMT):
So chain still receives msg from Propose api and call order?
yacovm (Mon, 16 Jul 2018 10:32:53 GMT):
I thought chain receives messages from `Broadcast`?
guoger (Mon, 16 Jul 2018 10:33:10 GMT):
that’s why I’m also swing between RaftCluster and RaftNode
guoger (Mon, 16 Jul 2018 10:33:39 GMT):
Leader also receives tx from follower nodes
yacovm (Mon, 16 Jul 2018 10:34:19 GMT):
cluster sounds good
guoger (Mon, 16 Jul 2018 10:36:49 GMT):
Btw, I may be running a late, super heavy rain today... roads around China lab are flooded...
guoger (Mon, 16 Jul 2018 11:49:53 GMT):
whenever you guys are ready
guoger (Mon, 16 Jul 2018 11:49:59 GMT):
@yacovm @C0rWin
C0rWin (Mon, 16 Jul 2018 12:40:27 GMT):
ready :)
yacovm (Mon, 16 Jul 2018 12:48:09 GMT):
....
yacovm (Mon, 16 Jul 2018 12:49:26 GMT):
@guoger ?
guoger (Mon, 16 Jul 2018 12:49:38 GMT):
k, just a sec
guoger (Mon, 16 Jul 2018 12:54:31 GMT):
https://zoom.us/j/359450646
yacovm (Mon, 16 Jul 2018 12:55:31 GMT):
@C0rWin
adarshsaraf123 (Mon, 16 Jul 2018 13:22:48 GMT):
Has joined the channel.
guoger (Mon, 16 Jul 2018 13:43:49 GMT):
just writing some notes here. We had a quick call, and agreed on how to split the work. We will come up with interfaces for each piece so we could independently work on mocks.
adarshsaraf123 (Mon, 16 Jul 2018 14:36:59 GMT):
Hi all, I am Adarsh Saraf working as a Software Engineer with IBM Research, India. I would like to contribute to the raft implementation. I have prior experience in porting the fabric-orderer code for a project we were working on internally at the India Research Lab. @kostas @C0rWin @yacovm @guoger
huikang (Tue, 17 Jul 2018 18:05:09 GMT):
Hi, where can I find the scrum schedule? I am interested in attending the all and see if I could contribute to the project. Thanks.
jyellick (Tue, 17 Jul 2018 19:25:52 GMT):
@huikang There is a general Fabric scrum that occurs MWF at 9:30 ET in #fabric-scrum. There's no scheduled scrum for orderer activities, instead, coordination typically happens in this channel, asynchronously
huikang (Tue, 17 Jul 2018 19:29:17 GMT):
@jyellick thanks for your answer. I will watch for the coordination in this channel.
guoger (Wed, 18 Jul 2018 02:37:00 GMT):
@adarshsaraf123 @huikang hi, we highly appreciate your interests in contributing to this work! we are working on the first mvp at this moment, which is estimated to be available by next week, and we will definitely have some `help-needed` items by then. Meanwhile, here are some materials you need to study:
design doc: https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?usp=sharing
work breakdown: https://docs.google.com/spreadsheets/d/1R02-xjdl4hNw90kN4NZ1kX0f4QoIYn51_1UHlwy-GQs/edit?usp=sharing
etcd/raft example: https://github.com/coreos/etcd/tree/master/contrib/raftexample
etcd/raft readme: https://github.com/coreos/etcd/blob/master/raft/README.md
guoger (Wed, 18 Jul 2018 02:45:46 GMT):
if you have any questions/doubts, feel free to post it here or in the doc
guoger (Thu, 19 Jul 2018 02:57:03 GMT):
question, how do we distinguish between `config block` and `normal block`? when our fsm wrapper `RaftCluster` has consented on a block, it sends it back to `chain` to be committed, and the `chain` needs to know if it's of type `config` or `normal` to decide to use either `WriteBlock` or `WriteConfigBlock`
adarshsaraf123 (Thu, 19 Jul 2018 06:28:15 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=bC7Aj5PdfbMrS7BaB) @guoger Currently, to my knowledge, there is no means to distinguish between config blocks and normal blocks. An approach would be to see the first transaction in the block to determine if it is a config transaction. Since config blocks contain only one transaction, the config transaction, this should suffice.
guoger (Thu, 19 Jul 2018 07:04:56 GMT):
yea... inspecting channel header is what we do in `broadcast`, however it sounds heavy in this case...
guoger (Thu, 19 Jul 2018 09:05:37 GMT):
not sure if it makes sense to add `type` to block metadata
```
message BlockMetadata {
enum Type {
NORMAL = 1;
CONFIG = 2;
}
Type type = 1;
repeated bytes metadata = 2;
}
```
guoger (Thu, 19 Jul 2018 09:07:18 GMT):
or, since it's only used internally in `chain`, we could simply wrap `Block` in a struct, which contains type field. Although, I feel this `type` is needed for other consensus type as well
guoger (Thu, 19 Jul 2018 09:07:18 GMT):
~or, since it's only used internally in `chain`, we could simply wrap `Block` in a struct, which contains type field. Although, I feel this `type` is needed for other consensus type as well~ this doesn't work... we need to preserve this info while it's being sent through raft
yacovm (Thu, 19 Jul 2018 09:37:44 GMT):
what do you need it for?
guoger (Thu, 19 Jul 2018 09:43:28 GMT):
when our fsm wrapper `RaftCluster` has consented on a block, it sends it back to `chain` to be committed, and the `chain` needs to know if it's of type `config` or `normal` to decide to use either `WriteBlock` or `WriteConfigBlock
guoger (Thu, 19 Jul 2018 09:43:28 GMT):
@yacovm when our fsm wrapper `RaftCluster` has consented on a block, it sends it back to `chain` to be committed, and the `chain` needs to know if it's of type `config` or `normal` to decide to use either `WriteBlock` or `WriteConfigBlock
guoger (Thu, 19 Jul 2018 09:43:28 GMT):
@yacovm when our fsm wrapper `RaftCluster` has consented on a block, it sends it back to `chain` to be committed, and the `chain` needs to know if it's of type `config` or `normal` to decide to use either `WriteBlock` or `WriteConfigBlock`
yacovm (Thu, 19 Jul 2018 09:45:47 GMT):
but... can't you just have 2 of these methods in the interface?
guoger (Thu, 19 Jul 2018 09:48:32 GMT):
how does that solve the problem? we still need to peel the block and inspect envelope given current proto, no?
guoger (Thu, 19 Jul 2018 09:48:32 GMT):
how does that solve the problem? we still need to peel the block and inspect envelope given current proto definition, no?
yacovm (Thu, 19 Jul 2018 09:49:25 GMT):
oh you're saying you don't know if the block that comes from the raft.Node is a config block or not?
guoger (Thu, 19 Jul 2018 09:51:53 GMT):
Right
yacovm (Thu, 19 Jul 2018 09:52:57 GMT):
can't we just inspect it?
yacovm (Thu, 19 Jul 2018 09:53:19 GMT):
if it has 1 envelope and the envelope contains a config update... doesn't that mean it's a config block?
guoger (Thu, 19 Jul 2018 10:32:15 GMT):
Yes, that’s what we do at the moment, in broadcast. Although I think it’s a bit heavy for every block
guoger (Thu, 19 Jul 2018 10:32:31 GMT):
I think it benefits us in the long run
yacovm (Thu, 19 Jul 2018 10:34:37 GMT):
Did you do benchmarking?
yacovm (Thu, 19 Jul 2018 10:35:26 GMT):
I say for now keep it like that and then we can change... we need to decide on some proto structures which we use to send around
yacovm (Thu, 19 Jul 2018 10:35:41 GMT):
right now the communication API that I write - just uses `proto.Message` which is the interface
yacovm (Thu, 19 Jul 2018 10:35:51 GMT):
because I want to make it (if possible) re-usable for BFT too
C0rWin (Thu, 19 Jul 2018 14:40:17 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=T7LfzX7BVyZWTpj0iF) @guoger well, there is already code for what Yacov proposed:
```// Processor provides the methods necessary to classify and process any message which
// arrives through the Broadcast interface.
type Processor interface {
// ClassifyMsg inspects the message header to determine which type of processing is necessary
ClassifyMsg(chdr *cb.ChannelHeader) Classification
// ProcessNormalMsg will check the validity of a message based on the current configuration. It returns the current
// configuration sequence number and nil on success, or an error if the message is not valid
ProcessNormalMsg(env *cb.Envelope) (configSeq uint64, err error)
// ProcessConfigUpdateMsg will attempt to apply the config update to the current configuration, and if successful
// return the resulting config message and the configSeq the config was computed from. If the config update message
// is invalid, an error is returned.
ProcessConfigUpdateMsg(env *cb.Envelope) (config *cb.Envelope, configSeq uint64, err error)
// ProcessConfigMsg takes message of type `ORDERER_TX` or `CONFIG`, unpack the ConfigUpdate envelope embedded
// in it, and call `ProcessConfigUpdateMsg` to produce new Config message of the same type as original message.
// This method is used to re-validate and reproduce config message, if it's deemed not to be valid anymore.
ProcessConfigMsg(env *cb.Envelope) (*cb.Envelope, uint64, error)
}```
guoger (Thu, 19 Jul 2018 15:09:35 GMT):
yup, I was aware of that (it's classifying envelope, not block though), just felt preserving this type in block could help us. Anyway, let's go with yacov proposed for now
sergefdrv (Mon, 23 Jul 2018 11:22:41 GMT):
minbft
sergefdrv (Mon, 23 Jul 2018 11:27:15 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=bozaAAWcsTtbMdQpe) I'm going to present this project Aug 2: https://goo.gl/LWAu8D. Please feed free to join if you're interested
sergefdrv (Mon, 23 Jul 2018 11:28:45 GMT):
@guoger ^^^
huikang (Tue, 24 Jul 2018 03:42:58 GMT):
Hi, quick question: where can I find the breakdown task list on the fabric jira? Thanks.
huikang (Tue, 24 Jul 2018 03:52:03 GMT):
In addition, I realized that the changes are spread into many small changes on gerrit, e.g., FAB-9449, FAB-9447. Is there a raft branch which includes these changes altogether to accommodate dev and testing?
adarshsaraf123 (Tue, 24 Jul 2018 06:40:26 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=zSrykGSkRvW93SsRC) @huikang This looks like a good breakdown but I don't find links in the same for the JIRA items: https://docs.google.com/spreadsheets/d/1R02-xjdl4hNw90kN4NZ1kX0f4QoIYn51_1UHlwy-GQs/edit#gid=0
adarshsaraf123 (Tue, 24 Jul 2018 06:40:26 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=zSrykGSkRvW93SsRC) @huikang This is the breakdown but not all the items have a corresponding JIRA: https://docs.google.com/spreadsheets/d/1R02-xjdl4hNw90kN4NZ1kX0f4QoIYn51_1UHlwy-GQs/edit#gid=0
huikang (Tue, 24 Jul 2018 15:06:31 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=d6xpyNbmkXLnAJ9pc) @adarshsaraf123 yes, I saw this as a pinned message. It would be nice if the jira ID can be added to the spreadsheet.
anarodrigues (Tue, 24 Jul 2018 18:53:32 GMT):
Has joined the channel.
kostas (Wed, 25 Jul 2018 14:10:14 GMT):
Hi everyone. I'll try to catch up with everything I've missed today and tomorrow.
kostas (Wed, 25 Jul 2018 16:01:20 GMT):
I'm currently going over recordings (maintainers meeting, JIRA changes, etc.).
kostas (Wed, 25 Jul 2018 16:02:17 GMT):
Would it be fair to say that the JIRA workflow hasn't been vastly improved and there is still a ridiculously amount of flexibility moving forward?
kostas (Wed, 25 Jul 2018 16:02:17 GMT):
Would it be fair to say that the JIRA workflow hasn't been vastly improved and there is still a ridiculous amount of flexibility moving forward?
kostas (Wed, 25 Jul 2018 16:02:17 GMT):
I see the marginal improvements in JIRA, but there is still a ridiculous amount of flexibility moving forward?
kostas (Wed, 25 Jul 2018 16:04:02 GMT):
Have we settled on one way of doing things?
kostas (Wed, 25 Jul 2018 16:04:02 GMT):
Have we settled on one way of doing things? (Whatever that way is.)
kostas (Wed, 25 Jul 2018 16:04:06 GMT):
(Whatever that way is.)
guoger (Wed, 25 Jul 2018 16:06:03 GMT):
I'll try to push my patch tomorrow..
kostas (Wed, 25 Jul 2018 16:06:30 GMT):
Jay, just to be clear -- I'm talking about some changes we're pushing to JIRA in general as Fabric team.
kostas (Wed, 25 Jul 2018 16:06:30 GMT):
Jay, just to be clear -- I'm talking about some changes we're pushing to JIRA in general across the project.
guoger (Wed, 25 Jul 2018 16:06:46 GMT):
one thing I'm currently struggling with is how to mock ticker so we have deterministic test
kostas (Wed, 25 Jul 2018 16:07:04 GMT):
None of the above relates to Raft. I still have a ton of catch up to do there.
kostas (Wed, 25 Jul 2018 18:04:15 GMT):
> I see the marginal improvements in JIRA, but there is still a ridiculous amount of flexibility moving forward?
As an example of this, when you bump into what is clearly a user-error submitted as bug, do you withdraw as Invalid: "Environment Issue", "Invalid: Works as Expected", or "Invalid: Test Error"?
kostas (Wed, 25 Jul 2018 18:04:15 GMT):
> I see the marginal improvements in JIRA, but there is still a ridiculous amount of flexibility moving forward?
As an example of this, when you bump into what is clearly a user-error submitted as a bug, do you withdraw as "Invalid: Environment Issue", "Invalid: Works as Expected", or "Invalid: Test Error"?
kostas (Wed, 25 Jul 2018 18:04:25 GMT):
I can make the case that all three are viable options.
kostas (Wed, 25 Jul 2018 20:59:19 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=rAEoPKjqh8RbRnEWf
kostas (Wed, 25 Jul 2018 20:59:35 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=rone4YMga8TzssrcW
kostas (Wed, 25 Jul 2018 21:00:11 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=kGBQcBr7fMkEvDFHG
kostas (Wed, 25 Jul 2018 21:03:42 GMT):
Rather confused about where the Raft work currently stands.
kostas (Wed, 25 Jul 2018 21:04:07 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=NHvvG72xf2cyx8KMD
kostas (Wed, 25 Jul 2018 21:04:24 GMT):
What was the verdict of the call?
kostas (Wed, 25 Jul 2018 21:04:51 GMT):
Let me know if there's code I could review to get a grasp of where things stand.
kostas (Wed, 25 Jul 2018 21:05:36 GMT):
I went through the messages that were exchanged, and I'm confused about the FSM approach.
yacovm (Wed, 25 Jul 2018 21:05:46 GMT):
I pushed a change set for comm layer, but I need to do some changes tomorrow
yacovm (Wed, 25 Jul 2018 21:05:55 GMT):
it's WIP
yacovm (Wed, 25 Jul 2018 21:06:12 GMT):
Kostas - it's much less simple than we thought ;)
kostas (Wed, 25 Jul 2018 21:06:12 GMT):
The node object that the etcd/raft library exposes is an FSM.
yacovm (Wed, 25 Jul 2018 21:06:24 GMT):
multi-channel TLS pinning isn't that simple ;)
yacovm (Wed, 25 Jul 2018 21:06:24 GMT):
multi-channel TLS pinning with RPCs (not streams!) isn't that simple ;)
kostas (Wed, 25 Jul 2018 21:07:23 GMT):
Ah, I'm interested in hearing more about this, when the time comes.
kostas (Wed, 25 Jul 2018 21:07:51 GMT):
Going back to the FSM comment, and at the risk of misinterpreting the earlier discussion, it would be a mistake to route _all_ messages via the raft.Node FSM.
kostas (Wed, 25 Jul 2018 21:08:09 GMT):
We only pass cut blocks to the FSM.
kostas (Wed, 25 Jul 2018 21:08:46 GMT):
_Unless_ you mean that we're building our own FSM around the raft.Node FSM, in which case the approach of "just pass everything to the FSM and we'll handle routing there" makes sense.
yacovm (Wed, 25 Jul 2018 21:08:49 GMT):
obviously... we weren't planning on routing all messages via it
yacovm (Wed, 25 Jul 2018 21:08:58 GMT):
where did you get that impression? :thinking:
kostas (Wed, 25 Jul 2018 21:09:05 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=kGBQcBr7fMkEvDFHG
yacovm (Wed, 25 Jul 2018 21:09:07 GMT):
oh
kostas (Wed, 25 Jul 2018 21:09:08 GMT):
^^
yacovm (Wed, 25 Jul 2018 21:09:13 GMT):
no, we talked since then
kostas (Wed, 25 Jul 2018 21:09:29 GMT):
Right, I figured I may be operating on stale data.
kostas (Wed, 25 Jul 2018 21:09:51 GMT):
At any rate, I guess I'll catch up one way or another.
kostas (Wed, 25 Jul 2018 21:10:06 GMT):
Feel free to tag me in upcoming CRs for review, or assign work my way.
kostas (Wed, 25 Jul 2018 21:10:25 GMT):
It'd be good to have a quick Zoom call with a couple of you to get up to speed as well, whenever you have the chance.
kostas (Wed, 25 Jul 2018 21:14:30 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=AZuAqE5GxYBzZAQa2
kostas (Wed, 25 Jul 2018 21:15:05 GMT):
@sergefdrv: I'll join that call, thanks for the tip. Would you have any interest in porting that work over to Fabric as well? (Building on top of the changes we'll build in for the Raft work.)
kostas (Wed, 25 Jul 2018 21:16:18 GMT):
@adarshsaraf123 @huikang: We'll try to straigthen things out by next week in JIRA. I'll keep you posted.
kostas (Thu, 26 Jul 2018 03:14:10 GMT):
> @sergefdrv: I'll join that call, thanks for the tip. Would you have any interest in porting that work over to Fabric as well? (Building on top of the changes we'll build in for the Raft work.)
Ah, nevermind that. I see the dependency on TEE.
adarshsaraf123 (Thu, 26 Jul 2018 03:56:26 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=buXk5Gq6RL2uYFfuk) @kostas I would also be interested in listening in on the call to understand the status.
kostas (Thu, 26 Jul 2018 04:07:17 GMT):
Right, if we are to have a call we’ll arrange it as-hoc in this very channel and also post the link here. We’ll @ you.
kostas (Thu, 26 Jul 2018 04:07:17 GMT):
Right, if we are to have a call we’ll arrange it ad-hoc in this very channel and also post the link here. We’ll @ you.
sergefdrv (Thu, 26 Jul 2018 08:38:58 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=3tD9o4Gxf9cQPro9x) @kostas I'm wondering, what's your position regarding consensus protocols utilizing TEE?
kostas (Thu, 26 Jul 2018 12:28:38 GMT):
> I'm wondering, what's your position regarding consensus protocols utilizing TEE?
@sergefdrv: Due to the TEE constraint, their applicability is quite limited.
kostas (Thu, 26 Jul 2018 12:28:38 GMT):
> I'm wondering, what's your position regarding consensus protocols utilizing TEE?
@sergefdrv Due to the TEE constraint, their applicability is quite limited.
sergefdrv (Thu, 26 Jul 2018 12:32:09 GMT):
@kostas I see your point, but TEE is becoming ubiquitous these days. Most modern PCs and servers are equipped with Intel SGX, whereas mobile devices have ARM TrustZone support...
kostas (Thu, 26 Jul 2018 12:35:24 GMT):
Ubiquitous is a bit of a stretch, but I also see your point. Here's this for another counterpoint: you have to trust Intel.
kostas (Thu, 26 Jul 2018 12:35:24 GMT):
Ubiquitous is a bit of a stretch, but let's see that popularity is not an issue. Here's this for another counterpoint: you have to trust Intel.
kostas (Thu, 26 Jul 2018 12:35:24 GMT):
Ubiquitous is a bit of a stretch, but let's assume that popularity is not an issue. Here's this for another counterpoint: you have to trust Intel.
sergefdrv (Thu, 26 Jul 2018 12:49:10 GMT):
@kostas I think it depends on a particular use case. My personal understanding is that it might be actually very hard to avoid a trusted third-party in a permissioned consortium completely, whether it's CA provider or TEE HW vendor
kostas (Thu, 26 Jul 2018 12:50:39 GMT):
Hm. I'd say it's several orders of magnitude easier to roll your own CA, than rolling your own TEE.
sergefdrv (Thu, 26 Jul 2018 12:54:07 GMT):
I mean, if we are fine to trust HTTPS certificates issued by well-known CA providers, why shouldn't we be fine trusting a TEE vendor? Of course, it depends on value of the asset at stake.
sergefdrv (Thu, 26 Jul 2018 12:54:51 GMT):
Anyways, it is only about ordering of transaction proposals, if I understand correctly
yacovm (Thu, 26 Jul 2018 12:55:14 GMT):
HTTPS certificates have expiration and there is certificate transparency ;)
yacovm (Thu, 26 Jul 2018 12:55:34 GMT):
Trusting hardware is much more problematic... take the bluetooth bug discovered this week
yacovm (Thu, 26 Jul 2018 12:55:56 GMT):
https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00128.html
yacovm (Thu, 26 Jul 2018 12:56:16 GMT):
> A vulnerability in Bluetooth® pairing potentially allows an attacker with physical proximity (within 30 meters) to gain unauthorized access via an adjacent network, intercept traffic and send forged pairing messages between two vulnerable Bluetooth® devices
yacovm (Thu, 26 Jul 2018 12:56:52 GMT):
Also - with all due respect... Fabric supports privately issued TLS certificates
sergefdrv (Thu, 26 Jul 2018 12:56:56 GMT):
remote attestation can also have expiration and revocation mechanisms. What do you mean exactly by "certificate transparency"?
yacovm (Thu, 26 Jul 2018 12:57:21 GMT):
google it ;)
yacovm (Thu, 26 Jul 2018 12:57:34 GMT):
https://www.certificate-transparency.org/
sergefdrv (Thu, 26 Jul 2018 12:57:35 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=WCN4q8LJzZAyBke2Z) @yacovm I see :slight_smile:
yacovm (Thu, 26 Jul 2018 12:58:54 GMT):
anyway - TEE is not a silver bullet....
yacovm (Thu, 26 Jul 2018 12:59:21 GMT):
also BFT isn't. it's just a model
sergefdrv (Thu, 26 Jul 2018 12:59:37 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=zPdsDqhxnsFNx6o5j) @yacovm Thanks, I'll look into this
sergefdrv (Thu, 26 Jul 2018 13:04:31 GMT):
suppose, a particular TEE utilized by BFT consensus protocol in Fabric orderer turns out to be completely broken. What would be the worst consequence?
yacovm (Thu, 26 Jul 2018 13:04:51 GMT):
chain fork
kostas (Thu, 26 Jul 2018 13:05:28 GMT):
I'd also add censorship to that list.
yacovm (Thu, 26 Jul 2018 13:05:38 GMT):
he asked the worst ;)
kostas (Thu, 26 Jul 2018 13:05:50 GMT):
Touché.
kostas (Thu, 26 Jul 2018 13:07:57 GMT):
Intel's license agreements reads:
kostas (Thu, 26 Jul 2018 13:08:01 GMT):
> You acknowledge and agree that Intel has the right to immediately suspend your use of an SGX enclave by removing your authorization from Intel’s WhiteList of authorized keys if: (i) you fail or if in good faith Intel believes that you fail to comply in any way with any of the requirements set forth in the SGX Licensee Guide; or (ii) Intel in its reasonable discretion determines it reasonable or necessary in order to comply with applicable laws or legal requests on INTEL CONFIDENTIAL Page 5 of 11 v160908; or (iii) Intel reasonably believes such suspension to be reasonably prudent or necessary.
kostas (Thu, 26 Jul 2018 13:08:27 GMT):
IANAL but this effectively means they can shut you down at any time?
sergefdrv (Thu, 26 Jul 2018 13:09:03 GMT):
I'm not so sure, if broken TEE used by MinBFT consensus in the ordering service could cause chain fork in Fabric validator nodes
sergefdrv (Thu, 26 Jul 2018 13:12:06 GMT):
liveness might be at question in case of disaster with TEE
yacovm (Thu, 26 Jul 2018 13:13:37 GMT):
why not?
yacovm (Thu, 26 Jul 2018 13:14:01 GMT):
I do't know what's MinBFT , by the way
yacovm (Thu, 26 Jul 2018 13:14:22 GMT):
I assumed you meant to ask - if you achieve Byzantine tolerance via running the orderer code in an enclave and you don't use BFT
sergefdrv (Thu, 26 Jul 2018 13:15:18 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=2D3uz5pwu7eQhuBWB) @yacovm because the validator nodes would need to get f+1 consistent replies from different ordering nodes
kostas (Thu, 26 Jul 2018 13:16:09 GMT):
We assume that F+1 enclosures are broken.
kostas (Thu, 26 Jul 2018 13:16:09 GMT):
We assume that f+1 enclosures are broken.
yacovm (Thu, 26 Jul 2018 13:16:09 GMT):
but if you run BFT why do you need TEE?
sergefdrv (Thu, 26 Jul 2018 13:16:54 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=H5qGZbjp3cDjgitjc) @yacovm it's not that easy. MinBFT doesn't run substantial part of consensus in TEE. It utilized TEE to guarantee that a malicious node cannot send conflicting consensus messages to different peers
yacovm (Thu, 26 Jul 2018 13:17:58 GMT):
so you need BFT anyway then, right?
sergefdrv (Thu, 26 Jul 2018 13:18:22 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=nPFWLSHy8N2eanNaa) @kostas I'm not sure that I got you right, but PBFT would break in case of f+1 faulty nodes as well
yacovm (Thu, 26 Jul 2018 13:18:54 GMT):
are you saying that minBFT passes some of the algorithm steps via TEE and because of this it's more efficient?
kostas (Thu, 26 Jul 2018 13:19:04 GMT):
@sergefdrv: I'm responding to this comment of yours here:
kostas (Thu, 26 Jul 2018 13:19:06 GMT):
> suppose, a particular TEE utilized by BFT consensus protocol in Fabric orderer turns out to be completely broken. What would be the worst consequence?
kostas (Thu, 26 Jul 2018 13:19:39 GMT):
And my response is: if f+1 participating enclaves are broken, then everything than can go wrong with a traditional PBFT scheme, can also go wrong with MinBFT.
sergefdrv (Thu, 26 Jul 2018 13:19:56 GMT):
We tried to summarize core concepts of MinBFT: https://github.com/nec-blockchain/minbft#concepts
kostas (Thu, 26 Jul 2018 13:19:58 GMT):
So you got this and this: https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=8i8TYa7GzmB3LxePo
sergefdrv (Thu, 26 Jul 2018 13:21:43 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=KnupEttuLrTRW3x72) @kostas by completely broken TEE I meant that any faulty replica can cheat on the part of consensus logic that must be protected by TEE
kostas (Thu, 26 Jul 2018 13:22:07 GMT):
I do not follow.
sergefdrv (Thu, 26 Jul 2018 13:25:00 GMT):
@kostas from which point? :slight_smile:
kostas (Thu, 26 Jul 2018 13:25:10 GMT):
> by completely broken TEE I meant that any faulty replica can cheat on the part of consensus logic that must be protected by TEE
kostas (Thu, 26 Jul 2018 13:25:40 GMT):
Just to speed up this conversation - what are we arguing here?
sergefdrv (Thu, 26 Jul 2018 13:25:53 GMT):
there is a part of consensus login protected by TEE. A faulty node is assumed to be unable to deviate in that part of protocol
sergefdrv (Thu, 26 Jul 2018 13:26:21 GMT):
broken TEE would mean that a faulty node could actually deviate
kostas (Thu, 26 Jul 2018 13:26:37 GMT):
And if you f+1 of those, the protocol is broken.
kostas (Thu, 26 Jul 2018 13:27:00 GMT):
And if you don't have f+1 of those, you're still crossing your fingers that Intel won't revoke your license.
sergefdrv (Thu, 26 Jul 2018 13:27:19 GMT):
the assumption that there is still at most f correct nodes is assumed to nevertheless hold
sergefdrv (Thu, 26 Jul 2018 13:27:19 GMT):
the assumption that there is still at most f faulty nodes is assumed to nevertheless hold
kostas (Thu, 26 Jul 2018 13:28:10 GMT):
This is where I don't follow. You asked what would happen if we have a broken TEE.
kostas (Thu, 26 Jul 2018 13:28:23 GMT):
And the response is simple: 1 broken TEE nothing, f+1 broken TEEs everything.
kostas (Thu, 26 Jul 2018 13:28:38 GMT):
If we're saying that we have at most f broken TEEs, this is OK.
kostas (Thu, 26 Jul 2018 13:28:50 GMT):
But the point we're tip-toeing around is:
kostas (Thu, 26 Jul 2018 13:28:55 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=8i8TYa7GzmB3LxePo
sergefdrv (Thu, 26 Jul 2018 13:29:29 GMT):
I mean, broken TEE would allow faulty replica to deviate from the part of consensus handled by TEE. The correct nodes would sill behave correctly and do not abuse TEE vulnerability
sergefdrv (Thu, 26 Jul 2018 13:32:05 GMT):
in case TEE vendor would disrupt your ordering service, the nodes can fall back to dummy unprotected substitute of TEE piece of the protocol. That would mean switching to CFT mode
sergefdrv (Thu, 26 Jul 2018 13:32:49 GMT):
I believe certain use cases would still be fine with this
sergefdrv (Thu, 26 Jul 2018 13:33:01 GMT):
there is always some risk
kostas (Thu, 26 Jul 2018 13:33:46 GMT):
> there is always some risk
Can we agree on this as an objectively true statement:
kostas (Thu, 26 Jul 2018 13:34:03 GMT):
Protocol A: A BFT protocol that depends on TEE.
kostas (Thu, 26 Jul 2018 13:34:03 GMT):
Protocol A: A BFT protocol that depends on SGX.
kostas (Thu, 26 Jul 2018 13:34:11 GMT):
Protocol B: A BFT protocol that does not depend on TEE.
kostas (Thu, 26 Jul 2018 13:34:11 GMT):
Protocol B: A BFT protocol that does not depend on SGX.
kostas (Thu, 26 Jul 2018 13:34:24 GMT):
Both protocols work correctly, i.e. no bugs.
kostas (Thu, 26 Jul 2018 13:34:40 GMT):
Then: Protocol A *always carries more risk* than protocol B.
kostas (Thu, 26 Jul 2018 13:34:58 GMT):
Actually, let me revise this statement.
sergefdrv (Thu, 26 Jul 2018 13:35:11 GMT):
I tend to personally agree with that
kostas (Thu, 26 Jul 2018 13:35:23 GMT):
I edited TEE to SGX.
sergefdrv (Thu, 26 Jul 2018 13:35:27 GMT):
why?
kostas (Thu, 26 Jul 2018 13:35:48 GMT):
Because: https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=8i8TYa7GzmB3LxePo
kostas (Thu, 26 Jul 2018 13:36:18 GMT):
Similar additional trust assumptions apply to any TEE, but I'd rather make a concrete statement here.
sergefdrv (Thu, 26 Jul 2018 13:36:22 GMT):
I mean, why we should speak about SGX rather than more generic term?
kostas (Thu, 26 Jul 2018 13:36:31 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=YmkqgCxs9LcFd8wyT
sergefdrv (Thu, 26 Jul 2018 13:36:32 GMT):
okay, I see
sergefdrv (Thu, 26 Jul 2018 13:37:30 GMT):
but at the same time Protocol A requires less computational resources and requires less consensus rounds
sergefdrv (Thu, 26 Jul 2018 13:37:56 GMT):
sound's like a trade-off, doesn't it?
sergefdrv (Thu, 26 Jul 2018 13:38:36 GMT):
and depending on a concrete use-case it might be reasonable to choose one or another option
kostas (Thu, 26 Jul 2018 13:39:28 GMT):
We're exchanging banal statements now :wink:
sergefdrv (Thu, 26 Jul 2018 13:40:11 GMT):
but if we forget about ups and downs of different approaches for a moment, I'd like to point to https://hyperledger-fabric.readthedocs.io/en/latest/whatis.html#pluggable-consensus
sergefdrv (Thu, 26 Jul 2018 13:40:34 GMT):
> multiple ordering services supporting different applications or application requirements
sergefdrv (Thu, 26 Jul 2018 13:40:59 GMT):
that's all about options and trade-offs, I believe :slight_smile:
kostas (Thu, 26 Jul 2018 13:41:53 GMT):
I'm just saying we're stating obvious things now.
kostas (Thu, 26 Jul 2018 13:42:38 GMT):
Yes, there are trade-offs. I never argue otherwise. You'll never find any semi-reasonable person arguing otherwise.
kostas (Thu, 26 Jul 2018 13:42:38 GMT):
Yes, there are trade-offs. I never argued otherwise. You'll never find any semi-reasonable person arguing otherwise.
sergefdrv (Thu, 26 Jul 2018 13:43:30 GMT):
that's why I'm wondering why it wouldn't make sense to try MinBFT as a Fabric ordering service
kostas (Thu, 26 Jul 2018 13:44:12 GMT):
By all means, go for it. We'd love to see this as an option.
sergefdrv (Thu, 26 Jul 2018 13:46:16 GMT):
:slight_smile:
sergefdrv (Thu, 26 Jul 2018 13:46:43 GMT):
We put it like:
> We hope that by evaluating this consensus component under the existing blockchain frameworks, the community will benefit from availability to leverage it in different practical use cases.
kostas (Fri, 27 Jul 2018 14:22:38 GMT):
Anybody using the new `#closes` convention in their CRs?
kostas (Fri, 27 Jul 2018 14:22:38 GMT):
Anybody using the new `#done` convention in their CRs?
kostas (Fri, 27 Jul 2018 14:22:46 GMT):
If so, please point me to one?
kostas (Sat, 28 Jul 2018 02:22:45 GMT):
https://chat.hyperledger.org/channel/fabric-scrum?msg=CuzWKSF7KepZfZiQK
kostas (Sat, 28 Jul 2018 12:44:30 GMT):
@jyellick: Just a heads up that I'll be slicing/dicing/restitching that config-related Raft work we had put out a couple of months ago into a different sequence early next week. Will message you to add your sign-off in anything that contains your code.
kostas (Sat, 28 Jul 2018 12:45:35 GMT):
@yacovm @guoger @C0rWin: Any chance we could have that sync-up call early next week? (Tuesday 8am EDT?)
yacovm (Sat, 28 Jul 2018 12:46:07 GMT):
Tuesday is a very good day for me
kostas (Sat, 28 Jul 2018 12:48:05 GMT):
I'd like to get started and help out with whatever stack you are working on. I have the config-related CRs above to work with, and some slides to prepare for the design review, so I should be good for a couple of days.
kostas (Sat, 28 Jul 2018 13:00:49 GMT):
Raft playback scheduled for this coming Thursday: https://wiki.hyperledger.org/projects/fabric/playbacks
C0rWin (Sun, 29 Jul 2018 09:39:04 GMT):
@kostas @yacovm Tuesday works for me as well
guoger (Sun, 29 Jul 2018 17:07:29 GMT):
I pushed a CR here: https://gerrit.hyperledger.org/r/#/c/24919/
however, it *only* meant to show interfaces I came up with, so we could have a discussion on top of it. I'll try to adapt to the CR from @yacovm , and coordinate with @C0rWin .
yacovm (Sun, 29 Jul 2018 17:19:34 GMT):
yeah, we can discuss tomorrow at length - but, basically - the comm layer is interfaced via 3 methods:
`Configure(channel string, members []*RemoteNode)` - configures the comm layer per channel. `RemoteNode` is `ID`, `Endpoint`, and server/client TLS certificates in DER encoding(NOT PEM!!!!)
`Remote(channel string, id uint64) (RemoteStub, error)` - returns a `RemoteStub` which is an abstraction over the `Step`/`Propose` stream/rpc.
`Handle(methodName, channel string, sender uint64, msg proto.Message) (proto.Message, error)` - whenever the comm layer receives a message, it calls this method and passes the channel, and the sender it identified using the TLS pinning, and of course - the message.
yacovm (Sun, 29 Jul 2018 17:19:34 GMT):
yeah, we can discuss tomorrow at length - but, basically - the comm layer is interfaced via 3 methods:
`Configure(channel string, members []*RemoteNode)` - configures the comm layer per channel. `RemoteNode` is `ID`, `Endpoint`, and server/client TLS certificates in DER encoding(NOT PEM!!!!)
`Remote(channel string, id uint64) (RemoteStub, error)` - returns a `RemoteStub` which is an abstraction over the `Step`/ `Propose` stream/rpc.
`Handle(methodName, channel string, sender uint64, msg proto.Message) (proto.Message, error)` - whenever the comm layer receives a message, it calls this method and passes the channel, and the sender it identified using the TLS pinning, and of course - the message.
yacovm (Sun, 29 Jul 2018 17:19:34 GMT):
yeah, we can discuss tomorrow or Tuesday at length - but, basically - the comm layer is interfaced via 3 methods:
`Configure(channel string, members []*RemoteNode)` - configures the comm layer per channel. `RemoteNode` is `ID`, `Endpoint`, and server/client TLS certificates in DER encoding(NOT PEM!!!!)
`Remote(channel string, id uint64) (RemoteStub, error)` - returns a `RemoteStub` which is an abstraction over the `Step`/ `Propose` stream/rpc.
`Handle(methodName, channel string, sender uint64, msg proto.Message) (proto.Message, error)` - whenever the comm layer receives a message, it calls this method and passes the channel, and the sender it identified using the TLS pinning, and of course - the message.
yacovm (Sun, 29 Jul 2018 17:19:34 GMT):
yeah, we can discuss tomorrow or Tuesday at length - but, basically - the comm layer is interfaced via 3 methods:
`Configure(channel string, members []*RemoteNode)` - configures the comm layer per channel. `RemoteNode` is `ID`, `Endpoint`, and server/client TLS certificates in DER encoding(NOT PEM!!!!)
`Remote(channel string, id uint64) (RemoteStub, error)` - returns a `RemoteStub` which is an abstraction over the `Step`/ `Propose` stream/rpc. Every operation with the `RemoteStub` would be sent over a connection to the appropriate remote node
`Handle(methodName, channel string, sender uint64, msg proto.Message) (proto.Message, error)` - whenever the comm layer receives a message, it calls this method and passes the channel, and the sender it identified using the TLS pinning, and of course - the message.
yacovm (Sun, 29 Jul 2018 17:19:34 GMT):
yeah, we can discuss tomorrow or Tuesday at length - but, basically - the comm layer is interfaced via 3 methods:
`Configure(channel string, members []*RemoteNode)` - configures the comm layer per channel. `RemoteNode` is `ID`, `Endpoint`, and server/client TLS certificates in DER encoding(NOT PEM!!!!)
`Remote(channel string, id uint64) (RemoteStub, error)` - returns a `RemoteStub` which is an abstraction over the `Step`/ `Propose` stream/rpc. Every operation with the `RemoteStub` would be sent over a shared connection to that node.
`Handle(methodName, channel string, sender uint64, msg proto.Message) (proto.Message, error)` - whenever the comm layer receives a message, it calls this method and passes the channel, and the sender it identified using the TLS pinning, and of course - the message.
yacovm (Sun, 29 Jul 2018 17:19:34 GMT):
yeah, we can discuss tomorrow or Tuesday at length - but, basically - the comm layer is interfaced via 3 methods:
`Configure(channel string, members []*RemoteNode)` - configures the comm layer per channel. `RemoteNode` is `ID`, `Endpoint`, and server/client TLS certificates in DER encoding(NOT PEM!!!!)
`Remote(channel string, id uint64) (RemoteStub, error)` - returns a `RemoteStub` which is an abstraction over the `Step`/ `Propose` stream/rpc.
Every operation with the `RemoteStub` would be sent over a shared connection to the remote node that has the server TLS certificate that was configured for the channel.
`Handle(methodName, channel string, sender uint64, msg proto.Message) (proto.Message, error)` - whenever the comm layer receives a message, it calls this method and passes the channel, and the sender it identified using the TLS pinning, and of course - the message.
yacovm (Sun, 29 Jul 2018 17:19:34 GMT):
yeah, we can discuss tomorrow or Tuesday at length - but, basically - the comm layer is interfaced via 3 methods:
`Configure(channel string, members []*RemoteNode)` - configures the comm layer per channel. `RemoteNode` is `ID`, `Endpoint`, and server/client TLS certificates in DER encoding(NOT PEM!!!!)
`Remote(channel string, id uint64) (RemoteStub, error)` - returns a `RemoteStub` which is an abstraction over the `Step`/ `Propose` stream/rpc.
Every operation with the `RemoteStub` would be sent over a shared connection to the remote node that has the server TLS certificate that was configured for the channel.
`Handle(methodName, channel string, sender uint64, msg proto.Message) (proto.Message, error)` - whenever the comm layer receives a message, it calls this method and passes the channel, and the sender it identified using the TLS pinning, and of course - the message. The implementer of the `Handle` will return a message as a response, and this message / error would be returned to the client
yacovm (Sun, 29 Jul 2018 17:20:57 GMT):
I am at the stage of writing UTs for the comm layer, and already have a basic test that passes with 2 nodes sending each other after `Configure()` and the `Handle` being dispatched correctly
yacovm (Sun, 29 Jul 2018 17:23:35 GMT):
I can add a `Shutdown()` method too
guoger (Mon, 30 Jul 2018 06:40:17 GMT):
@yacovm I'm trying to rebase my patch on top of yours. Need your advice on how to mock interfaces for testing fsm.
- should we define interfaces for `RPC`, so that `Step`, `SendPropose`, `ReceiveProposeResponse` could be mocked?
- on server side, not sure I want to mock at stream level. any better ideas?
yacovm (Mon, 30 Jul 2018 07:17:08 GMT):
That was the idea... but the RPC should be defined in your package @guoger
yacovm (Mon, 30 Jul 2018 07:17:08 GMT):
That was the idea... but the interface should be defined in your package @guoger
yacovm (Mon, 30 Jul 2018 07:17:55 GMT):
and you need to implement:
```
type Handler interface {
Handle(methodName, channel string, sender uint64, msg proto.Message) (proto.Message, error)
}
```
yacovm (Mon, 30 Jul 2018 07:18:36 GMT):
> - on server side, not sure I want to mock at stream level. any better ideas?
You don't need to mock a stream... just to implement the above method. The communication infrastructure does the stream handling on its own
guoger (Mon, 30 Jul 2018 07:36:50 GMT):
by mocking stream, I meant linking input/output of nodes without actually starting grpc server.
guoger (Mon, 30 Jul 2018 07:38:27 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=pCrrAYipp548Gbh7j) @yacovm not sure why these interfaces should be in my pkg? shouldn't it be part of communication?
yacovm (Mon, 30 Jul 2018 07:39:10 GMT):
> not sure why these interfaces should be in my pkg? shouldn't it be part of communication?
no... because you need it :)
sujanpgowda (Mon, 30 Jul 2018 14:45:00 GMT):
Has joined the channel.
kostas (Mon, 30 Jul 2018 15:01:12 GMT):
Just a reminder that we'll do a Zoom session tomorrow at 8am EDT to sync up: @yacovm @C0rWin @guoger /cc @muralisr @adarshsaraf123 @huikang @jeroiraz
kostas (Mon, 30 Jul 2018 15:01:31 GMT):
I'll post the link here a few minutes before the session is set to start.
yacovm (Mon, 30 Jul 2018 15:02:13 GMT):
Wait, how did Murali get into the list :O
kostas (Mon, 30 Jul 2018 15:03:23 GMT):
Murali reached out and asked to be in the loop and of course he's more than welcome to
yacovm (Mon, 30 Jul 2018 15:12:46 GMT):
@muralisr I see your taste in fabric nodes has matured
yacovm (Mon, 30 Jul 2018 15:13:10 GMT):
I always said the orderer looks sleeker than the peer
yacovm (Mon, 30 Jul 2018 15:13:19 GMT):
but you never listened
kostas (Tue, 31 Jul 2018 11:33:54 GMT):
Zoom call in 25 minutes from now.
kostas (Tue, 31 Jul 2018 11:33:54 GMT):
Zoom call in ~25~ 55 (see below) minutes from now.
kostas (Tue, 31 Jul 2018 11:43:41 GMT):
Artem has a conflict, so we'll have to push this back by 30 minutes. Sorry about the late notice. Updated the calendar invitation I sent your way.
kostas (Tue, 31 Jul 2018 12:28:17 GMT):
https://zoom.us/j/7432937602
guoger (Tue, 31 Jul 2018 13:09:07 GMT):
same linke?
kostas (Tue, 31 Jul 2018 13:09:13 GMT):
Yes.
kostas (Tue, 31 Jul 2018 18:10:29 GMT):
Following-up on today's call --
kostas (Tue, 31 Jul 2018 18:16:23 GMT):
We touched on the `rpc Step(raftpb.Message) returns (StepResponse)` definition being a not-so-good idea. Compilation when exposing `raftpb.Message` is broken (see: https://jira.hyperledger.org/browse/FAB-9864), that message alone is not enough as we need to piggyback the channel information (though we _could_ pass this in as grpc.Metadata no? this is also noted in the document), and it also seems like we can kill two birds with one stone here. `Step` and `Submit` are two methods that we will be using with any leader-based protocol, so we can make this implementation-agnostic.
kostas (Tue, 31 Jul 2018 18:19:50 GMT):
This is covered in: https://jira.hyperledger.org/browse/FAB-11412 and I'm on board with it.
kostas (Tue, 31 Jul 2018 18:22:40 GMT):
I'm still slightly lost on where https://jira.hyperledger.org/browse/FAB-11159 and https://jira.hyperledger.org/browse/FAB-11160 stand. @yacovm, you mentioned that their scope is subsumed by the work done in FAB-1161 I think, but if you can expand on it in JIRA that'd be great.
kostas (Tue, 31 Jul 2018 18:22:40 GMT):
I'm still slightly lost on where https://jira.hyperledger.org/browse/FAB-11159 and https://jira.hyperledger.org/browse/FAB-11160 stand. @yacovm, you mentioned that their scope is subsumed by the work done in FAB-11161 I think, but if you can expand on it in JIRA that'd be great.
yacovm (Tue, 31 Jul 2018 18:23:10 GMT):
i will in a bit
kostas (Tue, 31 Jul 2018 18:23:20 GMT):
No rush.
kostas (Tue, 31 Jul 2018 18:25:48 GMT):
@guoger: During the call today you talked about an integration test around my config-related work. Can you expand on it? What is the scenario you have in mind?
kostas (Tue, 31 Jul 2018 20:12:34 GMT):
I've updated the document to reflect the changes above.
guoger (Wed, 01 Aug 2018 06:29:05 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=TiqoK2x2gGqZGCBu8) @kostas I meant integration test would depend on config-related work. If we merge our works together by next week, we need as least one integration test to:
- start an OSN cluster using raft consensus
- send normal tx and expect blocks to be produced
guoger (Wed, 01 Aug 2018 06:30:05 GMT):
I see @yacovm is using mockery, do we have a decision on this? i'm currently using counterfeiter, but that could be changed
guoger (Wed, 01 Aug 2018 06:31:08 GMT):
also, why does `comm` dir contains pkg `consenster`?
yacovm (Wed, 01 Aug 2018 06:39:03 GMT):
Why do we need to be uniform?
yacovm (Wed, 01 Aug 2018 06:39:16 GMT):
I use whatever i want, you use whatever you want
yacovm (Wed, 01 Aug 2018 06:39:25 GMT):
This isn't an army 😉
yacovm (Wed, 01 Aug 2018 06:40:23 GMT):
@guoger because if it had been comm it would stutter
yacovm (Wed, 01 Aug 2018 06:40:44 GMT):
consenter.Communicator
yacovm (Wed, 01 Aug 2018 06:41:18 GMT):
I think after i'm done we should move that to orderer common
yacovm (Wed, 01 Aug 2018 06:41:27 GMT):
And not under raft
guoger (Wed, 01 Aug 2018 06:45:25 GMT):
to address comment https://gerrit.hyperledger.org/r/#/c/24919/1/protos/common/common.proto@181
> I do not see reason why we should add type into metadata field, you anyway have to unmarshal block to be able to access it, which is heaviest part. You can conclude block type from the header, no need to introduce such redundancy especially where impact or improvement is dubious.
how to conclude block type from header?
guoger (Wed, 01 Aug 2018 06:45:25 GMT):
to address comment https://gerrit.hyperledger.org/r/#/c/24919/1/protos/common/common.proto@181
> I do not see reason why we should add type into metadata field, you anyway have to unmarshal block to be able to access it, which is heaviest part. You can conclude block type from the header, no need to introduce such redundancy especially where impact or improvement is dubious.
how to conclude block type from header? @C0rWin
guoger (Wed, 01 Aug 2018 06:47:49 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=cce29c39-b31f-488e-a03b-3e5712106781) @yacovm agreed. Also, if we are to make
```
rpc Propose(stream ProposeRequest) returns (stream ProposeResponse);
rpc Step(StepRequest) returns (StepResponse);
```
reusable, it shouldn't be in `raft.proto` (probably communication.proto?)
also, let's change `common.Envelope content` to bytes, so consumer could load it with whatever is needed.
In raft case, we need config_seq here
guoger (Wed, 01 Aug 2018 06:47:49 GMT):
>I think after i'm done we should move that to orderer common
@yacovm agreed.
Also, if we are to make
```
rpc Propose(stream ProposeRequest) returns (stream ProposeResponse);
rpc Step(StepRequest) returns (StepResponse);
```
reusable, it shouldn't be in `raft.proto` (probably communication.proto?)
also, let's change `common.Envelope content` to bytes, so consumer could load it with whatever is needed.
In raft case, we need config_seq here
guoger (Wed, 01 Aug 2018 06:47:49 GMT):
>I think after i'm done we should move that to orderer common
@yacovm agreed.
Also, if we are to make
```
rpc Propose(stream ProposeRequest) returns (stream ProposeResponse);
rpc Step(StepRequest) returns (StepResponse);
```
reusable, it shouldn't be in `raft.proto` (probably communication.proto?)
also, let's change `common.Envelope content` in both `ProposeRequest` and `StepRequest` to bytes, so consumer could load it with whatever is needed.
In raft case, we need config_seq here
yacovm (Wed, 01 Aug 2018 07:44:13 GMT):
@guoger we merged the new protobuf schema
yacovm (Wed, 01 Aug 2018 07:44:26 GMT):
please use these messages :)
yacovm (Wed, 01 Aug 2018 07:44:37 GMT):
https://github.com/hyperledger/fabric/blob/master/protos/orderer/cluster.proto
guoger (Wed, 01 Aug 2018 07:45:45 GMT):
ah, I missed that, great
seokju.hong (Wed, 01 Aug 2018 12:52:44 GMT):
Has joined the channel.
kostas (Wed, 01 Aug 2018 14:18:38 GMT):
Let's tag everything with `raft` in JIRA to make sure we're not missing anything.
kostas (Wed, 01 Aug 2018 14:19:50 GMT):
Some of the text we have out there could be a bit more detailed, so I'll reach out to you with some questions and then expand it.
guoger (Wed, 01 Aug 2018 14:58:31 GMT):
imagine following scenario:
- nodeA is the leader, with follower nodeB, nodeC
- nodeA receives transactions, cache them in the blockcutter
- nodeA is isolated in the network, lost leadership
- nodeB is elected, start receiving transactions, cache them into its own blockcutter
- nodeA rejoins the network, cut the block, and `propose` it
I suppose, we should discard this block in this case?
guoger (Wed, 01 Aug 2018 14:58:31 GMT):
imagine following scenario:
- nodeA is the leader, with follower nodeB, nodeC
- nodeA receives transactions, cache them in the blockcutter
- nodeA is isolated in the network, lost leadership
- nodeB is elected, start receiving transactions, cache them into its own blockcutter
- nodeA *rejoins* the network, cuts the block, and `propose` it
I suppose, we should discard this block in this case?
yacovm (Wed, 01 Aug 2018 15:07:40 GMT):
i don't think this is your problem
yacovm (Wed, 01 Aug 2018 15:07:46 GMT):
raft should take care of it no?
seokju.hong (Wed, 01 Aug 2018 15:09:58 GMT):
hello :) the message of nodeA maybe discarded as i remember. if a message with *less term number* arrives, it is ignored
seokju.hong (Wed, 01 Aug 2018 15:10:25 GMT):
And here's quick question: how do you guys test/debug your own orderer?
yacovm (Wed, 01 Aug 2018 15:10:33 GMT):
we give it to uses, of course
yacovm (Wed, 01 Aug 2018 15:10:33 GMT):
we give it to users, of course
kostas (Wed, 01 Aug 2018 15:11:47 GMT):
@guoger: That's a good question. @seokju 1 got it right - the message will be ignored by the state machine.
kostas (Wed, 01 Aug 2018 15:14:08 GMT):
> And here's quick question: how do you guys test/debug your own orderer?
Mostly unit tests, and extended unit tests that are more like component-level integration tests.
kostas (Wed, 01 Aug 2018 15:15:25 GMT):
I want us to improve on this with the Raft work. I want to bring integration tests early on in the process.
seokju.hong (Wed, 01 Aug 2018 15:16:19 GMT):
Where could I find those unit tests?
yacovm (Wed, 01 Aug 2018 15:16:54 GMT):
once we have code merged we should have UTs
yacovm (Wed, 01 Aug 2018 15:16:57 GMT):
stay tuned
kostas (Wed, 01 Aug 2018 15:17:19 GMT):
Hold on - if you're talking solo or kafka, these are checked in already.
kostas (Wed, 01 Aug 2018 15:17:25 GMT):
Are you talking raft?
seokju.hong (Wed, 01 Aug 2018 15:17:39 GMT):
Yes, the raft one.
kostas (Wed, 01 Aug 2018 15:17:56 GMT):
What Yacov said then.
kostas (Wed, 01 Aug 2018 15:18:03 GMT):
Are you interested in contributing?
seokju.hong (Wed, 01 Aug 2018 15:18:44 GMT):
Of course :) but I cannot find where to start.
kostas (Wed, 01 Aug 2018 15:18:55 GMT):
@yacovm @C0rWin @guoger: I'll gladly draw it out if we don't have it already, but we need a shitty 5-minute diagram that connects the blocks that we're working on. We need to post this in JIRA.
kostas (Wed, 01 Aug 2018 15:19:26 GMT):
And we also need to document what the contract/interface between each of these blocks is.
kostas (Wed, 01 Aug 2018 15:20:46 GMT):
@seokju 1: I hear you. Give us until the end of the week to rectify this?
kostas (Wed, 01 Aug 2018 15:20:46 GMT):
@JaeseokLee : I hear you. Give us until the end of the week to rectify this?
kostas (Wed, 01 Aug 2018 15:20:46 GMT):
@seokju.hong: I hear you. Give us until the end of the week to rectify this?
seokju.hong (Wed, 01 Aug 2018 15:21:54 GMT):
OK, thanks!
guoger (Wed, 01 Aug 2018 16:00:24 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=5qEgjKJg5E9rEYQaH) @kostas i guess my follow-up question would be, how do we notify client? timeout?
kostas (Wed, 01 Aug 2018 17:15:07 GMT):
@guoger: What notification do you have in mind? Remember that the client figures out whether their transaction got accepted or not by inspecting the ledger. (See this discussion we had in the Google Doc regarding the submission guarantees: https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAAB62Ch5M)
kostas (Wed, 01 Aug 2018 17:15:27 GMT):
If I'm misunderstanding something, let me know.
huikang (Wed, 01 Aug 2018 20:56:58 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=7Nd8xchiWmihxo2c9) @yacovm Yes, raft takes care the proposals from nodeA. However, nodeA may need to notify the client who submits the transactions that the transactions will not be committed or resubmitted.
yacovm (Wed, 01 Aug 2018 20:57:37 GMT):
@huikang care to introduce yourself? :)
huikang (Wed, 01 Aug 2018 21:01:35 GMT):
I am working for IBM and based in New York. My previous open source experiences include Open Stack, docker, Open vswitch. I am familiar with the Raft and Paxos algorithm. I am new to Fabric and interested in the Raft-based orderer.
huikang (Wed, 01 Aug 2018 21:04:39 GMT):
I am looking at Jay's FSM implementation and see how I can contribute.
yacovm (Wed, 01 Aug 2018 21:12:59 GMT):
> Open Stack, docker, Open vswitch.
so you committed code to openstack, docker, and vswitch?
yacovm (Wed, 01 Aug 2018 21:13:03 GMT):
@huikang
huikang (Wed, 01 Aug 2018 21:13:46 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=PZCjPNCgMSmvhSb4x) @yacovm not too much.
huikang (Wed, 01 Aug 2018 21:15:04 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=PZCjPNCgMSmvhSb4x) @yacovm Are you IBMer? I saw a lot of fabric contributors are from IBM
yacovm (Wed, 01 Aug 2018 21:15:20 GMT):
openstack is in python isn't it? Are you familiar with Golang?
yacovm (Wed, 01 Aug 2018 21:15:26 GMT):
the fabric core is in Golang
huikang (Wed, 01 Aug 2018 21:15:43 GMT):
yep, I have been using Go for a while for some internal projects
yacovm (Thu, 02 Aug 2018 11:21:13 GMT):
@guoger @C0rWin @kostas I'm thinking instead of the RPC API that is now maybe it will be simpler to just have the comm layer expose directly:
`=!=ijiCQRpuiZ4AsW9c4=!=`
yacovm (Thu, 02 Aug 2018 11:21:13 GMT):
@guoger @C0rWin @kostas I'm thinking instead of the RPC API that is now maybe it will be simpler to just have the comm layer expose directly:
yacovm (Thu, 02 Aug 2018 11:21:40 GMT):
```
type Communicator interface {
// Remote returns a RemoteStub for the given RemoteNode ID in the context
// of the given channel, or error if connection cannot be established, or
// the channel wasn't configured
Remote(channel string, id uint64) (orderer.ClusterClient, error)
// Configure configures the communication to connect to all
// given members, and disconnect from any members not among the given
// members.
Configure(channel string, members []*RemoteNode)
// Shutdown shuts down the communicator
Shutdown()
}
```
yacovm (Thu, 02 Aug 2018 11:21:40 GMT):
```
type Communicator interface {
// Remote returns a RemoteStub for the given RemoteNode ID in the context
// of the given channel, or error if connection cannot be established, or
// the channel wasn't configured
Remote(channel string, id uint64) (orderer.ClusterClient, error)
}
```
yacovm (Thu, 02 Aug 2018 11:21:40 GMT):
```
type Communicator interface {
// Remote returns a RemoteStub for the given RemoteNode ID in the context
// of the given channel, or error if connection cannot be established, or
// the channel wasn't configured
Remote(channel string, id uint64) (ClusterClient, error)
}
```
yacovm (Thu, 02 Aug 2018 11:22:41 GMT):
when `ClusterClient` is
````
// Submit submits transactions to a cluster member
Submit(ctx context.Context, opts ...grpc.CallOption) (Cluster_SubmitClient, error)
// Step passes an implementation-specific message to another cluster member.
Step(ctx context.Context, in *StepRequest, opts ...grpc.CallOption) (*StepResponse, error)
```
without the context and grpc options
yacovm (Thu, 02 Aug 2018 11:22:41 GMT):
when `ClusterClient` is
````
// Submit submits transactions to a cluster member
Submit() (Cluster_SubmitClient, error)
// Step passes an implementation-specific message to another cluster member.
Step(req *StepRequest) (*StepResponse, error)
```
without the context and grpc options
yacovm (Thu, 02 Aug 2018 11:22:41 GMT):
when `ClusterClient` is
````
// Submit submits transactions to a cluster member
Submit() (Cluster_SubmitClient, error)
// Step passes an implementation-specific message to another cluster member.
Step(req *StepRequest) (*StepResponse, error)
```
joe-alewine (Thu, 02 Aug 2018 14:22:57 GMT):
Has joined the channel.
mhomaid (Thu, 02 Aug 2018 14:24:52 GMT):
Has joined the channel.
latitiah (Thu, 02 Aug 2018 14:50:32 GMT):
Has joined the channel.
kostas (Thu, 02 Aug 2018 14:54:38 GMT):
Is there a message you wanted to post here? (I see it ends in ":")
kostas (Thu, 02 Aug 2018 14:54:38 GMT):
@yacovm: Is there a follow-up to the message above?
zmaro (Fri, 03 Aug 2018 15:22:13 GMT):
Has joined the channel.
guoger (Sun, 05 Aug 2018 15:57:11 GMT):
submitted 3 CRs for review starting from https://gerrit.hyperledger.org/r/#/c/24919/
guoger (Sun, 05 Aug 2018 15:57:46 GMT):
i'll keep updating it to address comments received previously
guoger (Sun, 05 Aug 2018 15:59:06 GMT):
@C0rWin pls consult `RaftCluster` interface for a set of methods that you could use for your work
guoger (Sun, 05 Aug 2018 15:59:06 GMT):
@C0rWin pls consult `RaftCluster` interface for a set of methods that you could use for your work (https://gerrit.hyperledger.org/r/#/c/24919/2/orderer/consensus/raft/cluster.go)
yacovm (Sun, 05 Aug 2018 16:17:41 GMT):
@guoger why not squash all the 3 together? :thinking_face:
C0rWin (Sun, 05 Aug 2018 19:50:41 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=dfqdmTvPKBRQnFbNL) @guoger thanks
guoger (Mon, 06 Aug 2018 00:30:30 GMT):
I was told it was easier to review if split. But I certainly could squash @yacovm [ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=oMSZYWRNWvWgyC2kH)
guoger (Mon, 06 Aug 2018 07:40:17 GMT):
ok, this may be a very stupid question, but pls remind me, why is `Submit` rpc a bidi stream?
yacovm (Mon, 06 Aug 2018 07:45:56 GMT):
it's not a stupid question
yacovm (Mon, 06 Aug 2018 07:46:19 GMT):
I think that @kostas wanted it to look like the client's stream
guoger (Mon, 06 Aug 2018 08:04:29 GMT):
hmmm... but why?
also, for both `submit` and `step`, i'm not sure we actually care about the process result on the server. In another word, _from sender point of view, as long as the message is successfully delivered by comm, I don't need to know if server has processed it successfully, because I couldn't do anything about it anyway_
yacovm (Mon, 06 Aug 2018 08:06:21 GMT):
so, there is an API for the `raft.Node` that says that a specific node is un-healthy, no?
yacovm (Mon, 06 Aug 2018 08:06:27 GMT):
I'm not sure if we should use it
yacovm (Mon, 06 Aug 2018 08:06:37 GMT):
(thinking aloud here, again)
yacovm (Mon, 06 Aug 2018 08:06:42 GMT):
and i understand what you're saying
yacovm (Mon, 06 Aug 2018 08:08:16 GMT):
the code-review comments were more to stirr a discussion and point others to interesting points than criticism, @guoger
guoger (Mon, 06 Aug 2018 08:15:27 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=KdPASsshyMe8xaHdY) @yacovm which API is it?
guoger (Mon, 06 Aug 2018 08:16:08 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=jknnd9Dzdobbvr7qY) @yacovm happily accepting both criticism and discussion :)
yacovm (Mon, 06 Aug 2018 08:24:04 GMT):
`ReportUnreachable(id uint64)`
guoger (Mon, 06 Aug 2018 08:30:18 GMT):
oh, I thought meant an API to check if `raft.Node` is in erroneous state. You are certainly right, we could/should call this if _unreachable_ is reported by comm layer. but I'm not sure this should be called when the node received message, but throws error processing it.
guoger (Mon, 06 Aug 2018 08:30:18 GMT):
oh, I thought you meant an API to check if `raft.Node` is in erroneous state. You are certainly right, we could/should call this if _unreachable_ is reported by comm layer. but I'm not sure this should be called when the node received message, but throws error processing it.
yacovm (Mon, 06 Aug 2018 08:35:03 GMT):
i think that if we call it - we should call it when `Step()` returns an error
guoger (Mon, 06 Aug 2018 08:56:26 GMT):
seems that `Step()` error is not sent: https://github.com/coreos/etcd/blob/master/etcdserver/api/rafthttp/peer.go#L177-L182
yacovm (Mon, 06 Aug 2018 09:26:59 GMT):
@guoger but that doesn't prove that we shouldn't call that
yacovm (Mon, 06 Aug 2018 09:27:20 GMT):
I mean, what is that method for
yacovm (Mon, 06 Aug 2018 09:27:39 GMT):
also - isn't etcd in gRPC?
yacovm (Mon, 06 Aug 2018 09:27:56 GMT):
what is the use of rafthttp in etcd?
guoger (Mon, 06 Aug 2018 09:52:55 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=Crp7boHAwAtLcfYYA) @yacovm internal communication is over rafthttp
yacovm (Mon, 06 Aug 2018 09:53:21 GMT):
i see
yacovm (Mon, 06 Aug 2018 09:53:39 GMT):
well, up to you or @kostas i guess
yacovm (Mon, 06 Aug 2018 09:53:52 GMT):
now as for submit being a bidi-stream
yacovm (Mon, 06 Aug 2018 09:54:12 GMT):
i wanted it to be at least half a stream
yacovm (Mon, 06 Aug 2018 09:54:14 GMT):
or something
yacovm (Mon, 06 Aug 2018 09:54:39 GMT):
or an RPC
yacovm (Mon, 06 Aug 2018 09:55:13 GMT):
I don't mind personally what it will be
yacovm (Mon, 06 Aug 2018 09:55:28 GMT):
but if it's a bidi-stream then we should make use of it and send back a response no?
guoger (Mon, 06 Aug 2018 09:57:45 GMT):
but why _stream_?
guoger (Mon, 06 Aug 2018 09:59:18 GMT):
I don't have strong reason against it, i'm merely trying to infer the reason behind it, since I couldn't find one in design doc
yacovm (Mon, 06 Aug 2018 10:17:32 GMT):
because the client also has a stream
yacovm (Mon, 06 Aug 2018 10:17:39 GMT):
so i believe it was intended to mimic this
yacovm (Mon, 06 Aug 2018 10:17:55 GMT):
though I can't say I understand how that would work in practice
kostas (Mon, 06 Aug 2018 14:11:20 GMT):
There is indeed no concrete reason for anything being a stream.
kostas (Mon, 06 Aug 2018 14:11:48 GMT):
In the end, I did it mostly to maintain some sort of symmetry with AtomicBroadcast, which breaks of Step anyway.
kostas (Mon, 06 Aug 2018 14:13:08 GMT):
I did some digging into this when I was writing the doc, and as best as I can tell, the only downside to using a unary RPC is that every time you call a unary RPC you're creating a new HTTP/2 stream, and you run the risk of maxing out the number of concurrent streams a recipient allows.
kostas (Mon, 06 Aug 2018 14:13:14 GMT):
In our case however:
kostas (Mon, 06 Aug 2018 14:13:24 GMT):
1. We control the recipient, and
kostas (Mon, 06 Aug 2018 14:13:46 GMT):
2. In gRPC the default setting for `SETTINGS_MAX_CONCURRENT_STREAMS` is uncapped anyway
yacovm (Mon, 06 Aug 2018 14:13:49 GMT):
but the old one is deleted once the gRPC call ends, right?
yacovm (Mon, 06 Aug 2018 14:13:58 GMT):
it's not monotonously increasing, is it?
kostas (Mon, 06 Aug 2018 14:14:29 GMT):
Correct. I should probably edit the above to reflect that.
kostas (Mon, 06 Aug 2018 14:14:42 GMT):
This would be an issue for concurrent invocations then.
kostas (Mon, 06 Aug 2018 14:15:29 GMT):
Going back to the `ReportUnreachable` observation, I've noted this down in page 22.
kostas (Mon, 06 Aug 2018 14:16:21 GMT):
But my memory is fuzzy right now on the specifics of how it's used. `raftexample` and `swarmkit` should guide us here, and I can look this up if nobody's on it.
kostas (Mon, 06 Aug 2018 14:17:11 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=DS9FifiLAsZB5d4nK
kostas (Mon, 06 Aug 2018 14:17:17 GMT):
Expand on this Yacov?
guoger (Mon, 06 Aug 2018 14:17:39 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=tmHT4EJQ7fxfuEZMQ) @kostas if i get it correct, `raftexample` simply logs the error of `Step()`
guoger (Mon, 06 Aug 2018 14:17:49 GMT):
not sure about `swarmkit` though, I could look into it
kostas (Mon, 06 Aug 2018 14:18:12 GMT):
> But my memory is fuzzy right now on the specifics of how it's used. `raftexample` and `swarmkit` should guide us here, and I can look this up if nobody's on it.
From my handwritten notes on the subject:
kostas (Mon, 06 Aug 2018 14:19:18 GMT):
"`ReportUnreachable` and `ReportSnapshot` should be wired to the `raft.Node` implementation. They should be invoked when we are sending Raft messages to other nodes in the cluster."
kostas (Mon, 06 Aug 2018 14:19:22 GMT):
Yeah, not very helpful.
kostas (Mon, 06 Aug 2018 14:19:45 GMT):
Give `swarmkit` a look Jay, and I can give it a second look as well if need be.
kostas (Mon, 06 Aug 2018 23:17:12 GMT):
@jyellick: Have a look into the discussion over at: https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAACAcdGtU
kostas (Mon, 06 Aug 2018 23:17:46 GMT):
This touches heavily into the config framework, so I'd like your take on it. I'm hoping there is an easier solution that we're overlooking.
yacovm (Mon, 06 Aug 2018 23:29:24 GMT):
@jyellick ^
> Option 2, suggested by Yacov, is more complicated but allows each org to update the TLS certs of their OSN unilaterally. We move the "Consenter" message under the "Org" config group. An org will be able to define a ConfigValue called "Consenters", similar to how they define AnchorPeers today.
That's possible, right?
yacovm (Mon, 06 Aug 2018 23:29:35 GMT):
(just double-checking :) )
kostas (Mon, 06 Aug 2018 23:30:19 GMT):
(_If_ this is a question for me, I'm fairly certain the answer is yes, but this is where Jason comes in to confirm.)
kostas (Mon, 06 Aug 2018 23:30:19 GMT):
(_If_ this is a question for me, I've looked at the code and I'm fairly certain the answer is yes, but this is where Jason comes in to confirm.)
kostas (Mon, 06 Aug 2018 23:30:19 GMT):
(_If_ this is a question for me, I've looked at the code and I'm ~fairly~ rather certain the answer is yes, but this is where Jason comes in to confirm.)
guoger (Tue, 07 Aug 2018 06:09:56 GMT):
swarmkit simply logs `Step` error. `ReportUnreachable` is only invoked _iff_ transport layer has problem delivering the message. see https://github.com/docker/swarmkit/blob/master/manager/state/raft/raft.go?utf8=%E2%9C%93#L1454-L1456
cc @yacovm @kostas
guoger (Tue, 07 Aug 2018 06:09:56 GMT):
swarmkit _only_ logs `Step` error. `ReportUnreachable` is only invoked _iff_ transport layer has problem delivering the message. see https://github.com/docker/swarmkit/blob/master/manager/state/raft/raft.go?utf8=%E2%9C%93#L1454-L1456
cc @yacovm @kostas
guoger (Tue, 07 Aug 2018 06:09:56 GMT):
swarmkit _only_ logs `Step` error. `ReportUnreachable` is invoked _iff_ transport layer has problem delivering the message. see https://github.com/docker/swarmkit/blob/master/manager/state/raft/raft.go?utf8=%E2%9C%93#L1454-L1456
cc @yacovm @kostas
guoger (Tue, 07 Aug 2018 06:19:50 GMT):
there _may_ be a type of error returned by `Step`, for which we could call `ReportUnreachable` on the caller. But I think it's safe to simply ignore, rather than putting effort into finding that error type
jyellick (Tue, 07 Aug 2018 13:16:56 GMT):
> That's possible, right?
@yacovm Certainly, yes.
yacovm (Tue, 07 Aug 2018 13:17:38 GMT):
thanks, just wanted to make sure
kostas (Tue, 07 Aug 2018 19:00:54 GMT):
`make protos` on the latest master regenerates all (or almost all?) of the protobuf messages. Is this expected?
kostas (Tue, 07 Aug 2018 19:01:22 GMT):
I ran `make tools-docker-clean && make tools-docker` to make sure it's not an artifact of an older version running.
kostas (Tue, 07 Aug 2018 19:02:36 GMT):
Per Jason, a clean-all is needed to pick up the protobuf upgrade.
kostas (Tue, 07 Aug 2018 19:02:36 GMT):
(Update: Per Jason, a clean-all is likely needed to pick up the protobuf upgrade.)
yacovm (Tue, 07 Aug 2018 21:27:15 GMT):
```
type Handler interface {
Handle(methodName, channel string, sender uint64, msg proto.Message) (proto.Message, error)
}
```
Perhaps we should use here concrete types - the `SubmitRequest` and `StepRequest`, etc. etc. instead, and split `Handle()` to 2 methods - `OnStep` and `OnSubmit` ? @kostas @guoger @C0rWin ?
yacovm (Tue, 07 Aug 2018 21:28:04 GMT):
I think that it doesn't make sense to keep the polymorphism of the `proto.Message` given that we have the consensus-alg-agnostic protobuf schema now
kostas (Wed, 08 Aug 2018 00:19:28 GMT):
I agree.
kostas (Wed, 08 Aug 2018 00:57:16 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=FTLQ9Wvf3iDT3mRHo
kostas (Wed, 08 Aug 2018 01:03:37 GMT):
@C0rWin: This should allow you to build on top hopefully: https://gerrit.hyperledger.org/r/c/25235/
kostas (Wed, 08 Aug 2018 01:04:00 GMT):
And as we discussed earlier today, the contract is identical to the previous merge-conflicted CRs that were out there.
kostas (Wed, 08 Aug 2018 01:09:04 GMT):
@yacovm: Your work does not touch on the dispatcher box shown here, correct? https://docs.google.com/presentation/d/1olukZUnriPS3HfWMPkQfKjypa3QV_iTacfI1lH8Mtqg/edit#slide=id.g3dec6414ec_0_0
kostas (Wed, 08 Aug 2018 01:09:04 GMT):
@yacovm: You are not working on the dispatcher box shown here, correct? https://docs.google.com/presentation/d/1olukZUnriPS3HfWMPkQfKjypa3QV_iTacfI1lH8Mtqg/edit#slide=id.g3dec6414ec_0_0
kostas (Wed, 08 Aug 2018 01:12:58 GMT):
(Looking to write a sub-task for it.)
guoger (Wed, 08 Aug 2018 02:46:24 GMT):
are we already using `zap` in our codebase?
huikang (Wed, 08 Aug 2018 02:59:51 GMT):
Hi, a quick question about submitting block to the raft.fsm. Is it only the leader OSN (assuming all OSN participates in the raft cluster) 's blockcutter can submit block to the fsm?
guoger (Wed, 08 Aug 2018 03:48:40 GMT):
@huikang yes
huikang (Wed, 08 Aug 2018 03:50:53 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=3Mrv646veqmHokLvT) @guoger thanks
guoger (Wed, 08 Aug 2018 06:21:53 GMT):
what's the purpose of this method and how's that not achieved by `Commit` in current implementation? https://docs.google.com/presentation/d/1olukZUnriPS3HfWMPkQfKjypa3QV_iTacfI1lH8Mtqg/edit?disco=AAAABzkT3PA cc @kostas
guoger (Wed, 08 Aug 2018 06:44:31 GMT):
and I'm having hard time understanding this:
> The rough rule of thumb should be: export EVERYTHING (the struct *and* its members). Don't think in terms of APIs and contracts that we have with the user.
structs should be exported, I agree. However, skim through stdlib, most of stuct members are hidden. I'd really appreciate some more verbose explanations. thx
guoger (Wed, 08 Aug 2018 06:44:31 GMT):
and I'm having hard time understanding this:
> The rough rule of thumb should be: export EVERYTHING (the struct *and* its members). Don't think in terms of APIs and contracts that we have with the user.
(https://gerrit.hyperledger.org/r/c/24919/6/orderer/consensus/raft/cluster.go#47)
structs should be exported, I agree. However, skim through stdlib, most of stuct members are hidden. I'd really appreciate some more verbose explanations. thx
yacovm (Wed, 08 Aug 2018 07:51:00 GMT):
I agree with @guoger
yacovm (Wed, 08 Aug 2018 07:51:23 GMT):
and you (Jay) didn't tell what you think about:
yacovm (Wed, 08 Aug 2018 07:51:30 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=XdAgBurkxHQzdemAe
guoger (Wed, 08 Aug 2018 07:56:17 GMT):
like this?
```
type Handler interface {
OnStep(req *StepRequest, sender uint64) (res *StepResponse, error)
OnSubmit(req *SubmitRequest, sender uint64) (res *SubmitResponse, error)
}
```
guoger (Wed, 08 Aug 2018 07:56:17 GMT):
like this? @yacovm
```
type Handler interface {
OnStep(req *StepRequest, sender uint64) (res *StepResponse, error)
OnSubmit(req *SubmitRequest, sender uint64) (res *SubmitResponse, error)
}
```
yacovm (Wed, 08 Aug 2018 08:01:58 GMT):
yeah
guoger (Wed, 08 Aug 2018 08:07:56 GMT):
which component is supposed to implement this interface? https://docs.google.com/presentation/d/1olukZUnriPS3HfWMPkQfKjypa3QV_iTacfI1lH8Mtqg/edit?usp=sharing
guoger (Wed, 08 Aug 2018 08:09:00 GMT):
i suppose `dispatcher` in the diagram?
yacovm (Wed, 08 Aug 2018 08:10:17 GMT):
which one?
guoger (Wed, 08 Aug 2018 08:11:14 GMT):
`dispatcher` in `etcdraftplugin` at bottom
yacovm (Wed, 08 Aug 2018 08:11:49 GMT):
so I think if we split the `Handle` to 2 methods
yacovm (Wed, 08 Aug 2018 08:11:52 GMT):
we'll have 2 dispatchers
yacovm (Wed, 08 Aug 2018 08:11:54 GMT):
no?
adarshsaraf123 (Wed, 08 Aug 2018 08:19:32 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=5xN6KqMLL4jEvz7o9) @guoger Shouldn't we be adding the channel name as a parameter as well?
guoger (Wed, 08 Aug 2018 08:21:03 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=2MjahaLqjaeR5Aqer) @yacovm I think I need a walkthrough of message flow...
guoger (Wed, 08 Aug 2018 08:21:28 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=WTZAHWb6d2mnqaNoA) @adarshsaraf123 if channel is included in both `StepReq` and `SubmitReq`, why should we?
guoger (Wed, 08 Aug 2018 08:22:19 GMT):
in `cluster.proto`
```
message StepRequest {
string channel = 1;
bytes payload = 2;
}
// SubmitRequest wraps a transaction to be sent for ordering
message SubmitRequest {
string channel = 1;
// last_validation_seq denotes the last
// configuration sequence at which the
// sender validated this message
uint64 last_validation_seq = 2;
// content is the fabric transaction
// that is forwarded to the cluster member
common.Envelope content = 3;
}
```
adarshsaraf123 (Wed, 08 Aug 2018 08:22:43 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=wYmQGSEKNxzKeZ4WF) @guoger :+1_tone4:
guoger (Wed, 08 Aug 2018 08:23:51 GMT):
i started missing 'thread' in slack :P
yacovm (Wed, 08 Aug 2018 08:28:09 GMT):
@guoger what do you mean walkthrough?
yacovm (Wed, 08 Aug 2018 08:28:20 GMT):
right now, someone needs to implement the `Handle` method, right?
guoger (Wed, 08 Aug 2018 08:30:36 GMT):
mainly looking for:
- who's implementing `ClusterServer` interface in grpc?
- who calls dispatcher?
- dispatcher looks up targeted `Chain`, and call `Order/Configure/Step`, correct?
guoger (Wed, 08 Aug 2018 08:30:36 GMT):
mainly looking for:
- ~who~which component is implementing `ClusterServer` interface in grpc?
- ~who~which component calls dispatcher?
- dispatcher looks up targeted `Chain`, and call `Order/Configure/Step`, correct?
guoger (Wed, 08 Aug 2018 08:31:40 GMT):
(by who, I mean component, not person)
yacovm (Wed, 08 Aug 2018 08:31:42 GMT):
1) Me
2) What's dispatcher? right now we have:
```
type Handler interface {
Handle(methodName, channel string, sender uint64, msg proto.Message) (proto.Message, error)
}
```
guoger (Wed, 08 Aug 2018 08:36:23 GMT):
@yacovm edited my questions, pardon my grammar :P
yacovm (Wed, 08 Aug 2018 08:37:12 GMT):
1) the `service.go` in my c hange set
2) I guess... some components needs to dispatch the function calls to the right chain. I am not implementing it at this time
yacovm (Wed, 08 Aug 2018 08:37:39 GMT):
and I don't know which component, but it should be pretty trivial no?
guoger (Wed, 08 Aug 2018 08:43:27 GMT):
I agree with `OnSubmit`/`OnStep` :)
guoger (Wed, 08 Aug 2018 08:43:27 GMT):
I agree with `OnSubmit` / `OnStep` :)
adarshsaraf123 (Wed, 08 Aug 2018 09:04:41 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=PatgSwDpCMKa9xDTh) @yacovm The component handling (2) has been named dispatcher in this [diagram](https://docs.google.com/presentation/d/1olukZUnriPS3HfWMPkQfKjypa3QV_iTacfI1lH8Mtqg/edit?disco=AAAABzkT3PE).
yacovm (Wed, 08 Aug 2018 09:08:07 GMT):
@adarshsaraf123 thanks
adarshsaraf123 (Wed, 08 Aug 2018 09:12:17 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=HenM5iysYbgXBc5Fs) @kostas @yacovm @kostas had asked you about this. If you aren't intending to work on this, I was planning on volunteering for the same.
adarshsaraf123 (Wed, 08 Aug 2018 09:12:17 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=HenM5iysYbgXBc5Fs) @kostas @yacovm, Kostas had asked you about this. If you aren't intending to work on this, I was planning on volunteering for the same.
yacovm (Wed, 08 Aug 2018 09:45:24 GMT):
I said i am not working on this at the moment
yacovm (Wed, 08 Aug 2018 09:46:09 GMT):
But before anyone does any work for this, I guess @guoger needs to change his code to adjust to the `OnSubmit` and `OnStep` no?
guoger (Wed, 08 Aug 2018 09:49:10 GMT):
`dispatcher` should be interacting with `Chain`, so it's up to @C0rWin
yacovm (Wed, 08 Aug 2018 10:02:12 GMT):
I guess
kostas (Wed, 08 Aug 2018 10:04:52 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=wudhu247vxthxtsGF
kostas (Wed, 08 Aug 2018 10:05:28 GMT):
@guoger: What is/was the purpose of `sendChan`? That's exactly the purpose of this method?
kostas (Wed, 08 Aug 2018 10:05:28 GMT):
@guoger: What is/was the purpose of `sendChan`? I would argue then that this method serves the same purpose? I may be missing something.
kostas (Wed, 08 Aug 2018 10:06:22 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=zqMm2HxKPBBQJ9SHs
kostas (Wed, 08 Aug 2018 10:27:20 GMT):
The next two sentences in that Gerrit comment go over the why. By locking/hiding things down, you make wiring/setting/testing needlessly difficult. We have been traditionally very bad at making our dependencies explicit and at nailing down how one should interact with our structs. Add to that the fact that we're short-sighted when it comes to anticipating who else might wish to interact with our components (esp. when it comes to testing). I don't think Matt's presentation is recorded anywhere; I'll see if there are any posts out there that make a more eloquent and convincing case for this.
kostas (Wed, 08 Aug 2018 10:34:21 GMT):
Or @sykesm maybe you: (1) have such references handy, (2) can make the case for this more eloquently than I can, or (3) will correct me if any of the above reads wrong. (For reference, we started from here: https://gerrit.hyperledger.org/r/c/24919/6/orderer/consensus/raft/cluster.go#47)
guoger (Thu, 09 Aug 2018 03:57:31 GMT):
not a very authoritative source of information, but https://www.reddit.com/r/golang/comments/3ia88a/exported_unexported_best_practices/
guoger (Thu, 09 Aug 2018 03:57:31 GMT):
not a very authoritative source of information, but just using it for the sake of arguments https://www.reddit.com/r/golang/comments/3ia88a/exported_unexported_best_practices/
guoger (Thu, 09 Aug 2018 03:58:39 GMT):
> definitely don't export everything by default. In fact, I'd argue the opposite: export only the bare minimum required, and think hard about anything else you export beyond that. Every exported symbol is weight added to your API, and risk of API breakage if you have to change it.
jyellick (Thu, 09 Aug 2018 03:59:25 GMT):
I think this only applies to _API_, not to internal packages
jyellick (Thu, 09 Aug 2018 04:00:14 GMT):
In general, no one else consumes any of this fabric code, we could (and probably should) put it into an `internal` package to keep other packages from creating dependencies on exported symbols that are not intended to be API
jyellick (Thu, 09 Aug 2018 04:01:34 GMT):
https://golang.org/doc/go1.4#internalpackages
guoger (Thu, 09 Aug 2018 04:10:58 GMT):
> we could (and probably should) put it into an `internal` package
agreed. But still, I don't think _hiding struct members in internal package_ is anti-pattern
guoger (Thu, 09 Aug 2018 04:11:47 GMT):
I feel the opposite by looking at _internal_ packages in stdlib
guoger (Thu, 09 Aug 2018 04:11:47 GMT):
I feel the opposite by looking at _internal_ packages in stdlib (obviously i didn't exhaust the entire lib, but just a quick scan)
guoger (Thu, 09 Aug 2018 04:21:10 GMT):
@jyellick I see you are typing and I know it's quite late there. But I do need some food at the moment. I'll read your message later and respond
guoger (Thu, 09 Aug 2018 04:21:48 GMT):
thanks for having this discussion with me btw
jyellick (Thu, 09 Aug 2018 04:39:12 GMT):
I was a bit skeptical at first, I used to hide everything by default, but I've tried switching to the pattern of exporting everything, unless there is a good reason not to. It has made wiring and unit and integration tests much simpler.
There are also some good very simple reasons to do so for the purposes of mocking. It seems to be a fairly well established and good pattern not to mix mock files into the package directory and to instead isolated them into a `mock` package. Consider an interface defined in a package which returns a struct, also defined in that package. To generate the mock, satisfying the interface, the mock must import the struct. This causes a bit of a predicament, because the mock must import the package, and the test wishes to import the mock. The only real way to bypass this is to test from outside the package, (usually by declaring the test file to be in the `
jyellick (Thu, 09 Aug 2018 04:39:12 GMT):
I was a bit skeptical at first, I used to hide everything by default, but I've tried switching to the pattern of exporting everything, unless there is a good reason not to. It has made wiring and unit and integration tests much simpler.
There are also some good very simple reasons to do so for the purposes of mocking. It seems to be a fairly well established and good pattern not to mix mock files into the package directory and to instead isolated them into a `mock` package. Consider an interface defined in a package which returns a struct, also defined in that package. To generate the mock, satisfying the interface, the mock must import the struct. This causes a bit of a predicament, because the mock must import the package, and the test wishes to import the mock. The only real way to bypass this is to test from outside the package, (usually by declaring the test file to be in the `
jyellick (Thu, 09 Aug 2018 04:39:12 GMT):
I was a bit skeptical at first, I used to hide everything by default, but I've tried switching to the pattern of exporting everything, unless there is a good reason not to. It has made wiring and unit and integration tests much simpler, and I'm generally becoming a fan.
There are also some good very simple reasons to do so for the purposes of mocking. It seems to be a fairly well established and good pattern not to mix mock files into the package directory and to instead isolated them into a `mock` package. Consider an interface defined in a package which returns a struct, also defined in that package. To generate the mock, satisfying the interface, the mock must import the struct. This causes a bit of a predicament, because the mock must import the package, and the test wishes to import the mock. The only real way to bypass this is to test from outside the package, (usually by declaring the test file to be in the `
jyellick (Thu, 09 Aug 2018 04:39:12 GMT):
I was a bit skeptical at first, I used to hide everything by default, but I've tried switching to the pattern of exporting everything, unless there is a good reason not to. It has made wiring and unit and integration tests much simpler, and I'm generally becoming a fan.
There are also some good very simple reasons to do so for the purposes of mocking. It seems to be a fairly well established and good pattern not to mix mock files into the package directory and to instead isolated them into a `mock` package. Consider an interface defined in a package which returns a struct, also defined in that package. To generate the mock, satisfying the interface, the mock must import the struct. This causes a bit of a predicament, because the mock must import the package, and the test wishes to import the mock (resulting in an import cycle). The only real way to bypass this is to test from outside the package, (usually by declaring the test file to be in the `
guoger (Thu, 09 Aug 2018 06:52:44 GMT):
> test from outside the package
This is very true, and that's why `ginkgo` already generates tests in `
guoger (Thu, 09 Aug 2018 06:52:44 GMT):
> test from outside the package
This is very true, and that's why `ginkgo` already generates tests in `
guoger (Thu, 09 Aug 2018 06:52:44 GMT):
> test from outside the package
This is very true, and that's why `ginkgo` already generates tests in `
guoger (Thu, 09 Aug 2018 06:52:44 GMT):
> test from outside the package
This is very true, and that's why `ginkgo` already generates tests in `
guoger (Thu, 09 Aug 2018 06:52:44 GMT):
> test from outside the package
This is very true, and that's why `ginkgo` already generates tests in `
guoger (Thu, 09 Aug 2018 06:52:44 GMT):
> test from outside the package
This is very true, and that's why `ginkgo` already generates tests in `
guoger (Thu, 09 Aug 2018 06:52:44 GMT):
> test from outside the package
This is very true, and that's why `ginkgo` already generates tests in `
guoger (Thu, 09 Aug 2018 06:52:44 GMT):
> test from outside the package
This is very true, and that's why `ginkgo` already generates tests in `
guoger (Thu, 09 Aug 2018 06:52:44 GMT):
> test from outside the package
This is very true, and that's why `ginkgo` already generates tests in `
guoger (Thu, 09 Aug 2018 06:52:44 GMT):
> test from outside the package
This is very true, and that's why `ginkgo` already generates tests in `
guoger (Thu, 09 Aug 2018 06:52:44 GMT):
> test from outside the package
This is very true, and that's why `ginkgo` already generates tests in `
guoger (Thu, 09 Aug 2018 06:52:44 GMT):
> test from outside the package
This is very true, and that's why `ginkgo` already generates tests in `
guoger (Thu, 09 Aug 2018 06:52:44 GMT):
> test from outside the package
This is very true, and that's why `ginkgo` already generates tests in `
guoger (Thu, 09 Aug 2018 06:52:44 GMT):
> test from outside the package
This is very true, and that's why `ginkgo` already generates tests in `
sykesm (Thu, 09 Aug 2018 20:32:20 GMT):
So, any consistently applied philosophy can work. That said, what we have in fabric right now does not. By advocating for exported concrete types with exported fields for dependencies, it enables easier wiring and reuse. That's been demonstrated. And yes, it has some cost.
Your constructor example, while fine, could just as easily be implemented with a drivable() function instead of an attribute to achieve the same result. If the Drivable attribute is something consumers care about, then it would be exported. So, in general, that kind of logic usually raises flags for me.
Again, it's not
So, I don't advocate for exporting *everything* - I advocate for exporting injection points for dependencies and configuration. I also push people towards considering why something is or isn't exported; I simply don't believe the export-as-a-last resort philosophy for package we do not intent to be external API.
For example, I see many places where a sync.Mutex is embedded in a struct so (lazy) programmers can simply call lock on the self-reference without naming the lock. This exposes Lock and Unlock on the struct. I'd rarely (if ever) expect external lock management to be intended yet, we have it.
I also advocate for using _test packages to force authors to consider how the api surface will be used and what the capabilities are. If dependencies are structured properly, we can still get to very high code coverage. If we need tests in the main production package, so be it - but I don't believe that's the unilateral starting point.
As for internal packages, I already have a JIRA opened for that. I strongly believe we should restructure all of our packages and move the vast majority into an internal namespace.
sykesm (Thu, 09 Aug 2018 20:32:20 GMT):
So, any consistently applied philosophy can work. That said, what we have in fabric right now does not. By advocating for exported concrete types with exported fields for dependencies, it enables easier wiring and reuse. That's been demonstrated. And yes, it has some cost.
Your constructor example, while fine, could just as easily be implemented with a drivable() function instead of an attribute to achieve the same result. If the Drivable attribute is something consumers care about, then it would be exported. So, in general, that kind of logic usually raises flags for me.
In general, I think constructors are overused. In general, I think factories are overused. I know singletons are overused and abused in fabric - especially when it comes to factories and config.
Again, it's not like we never want or need a constructor. Where we need one, we need one - commonly when we have to instantiate maps and channels another other internals. When we do have one, we should consider how it evolves. The option function pattern and config structs (a dependency) should be considered and employed. Long arg lists on construction should not.
So, I don't advocate for exporting *everything* - I advocate for exporting injection points for dependencies and configuration. I also push people towards considering why something is or isn't exported; I simply don't believe the export-as-a-last resort philosophy for package we do not intent to be external API.
For example, I see many places where a sync.Mutex is embedded in a struct so (lazy) programmers can simply call lock on the self-reference without naming the lock. This exposes Lock and Unlock on the struct. I'd rarely (if ever) expect external lock management to be intended yet, we have it.
I also advocate for using _test packages to force authors to consider how the api surface will be used and what the capabilities are. If dependencies are structured properly, we can still get to very high code coverage. If we need tests in the main production package, so be it - but I don't believe that's the unilateral starting point.
As for internal packages, I already have a JIRA opened for that. I strongly believe we should restructure all of our packages and move the vast majority into an internal namespace.
sykesm (Thu, 09 Aug 2018 20:32:20 GMT):
So, any consistently applied philosophy can work. That said, what we have in fabric right now does not. By advocating for exported concrete types with exported fields for dependencies, it enables easier wiring and reuse. That's been demonstrated. And yes, it has some cost.
Your constructor example, while fine, could just as easily be implemented with a drivable() function instead of an attribute to achieve the same result. If the Drivable attribute is something consumers care about, then it would be exported. So, in general, that kind of logic usually raises flags for me.
In general, I think constructors are overused. In general, I think factories are overused. I know singletons are overused and abused in fabric - especially when it comes to factories and config.
Again, it's not like we never want or need a constructor. Where we need one, we need one - commonly when we have to instantiate maps and channels another other internals. When we do have one, we should consider how it evolves. The option function pattern and config structs (a dependency) should be considered and employed. Long arg lists on construction should not.
So, I don't advocate for exporting *everything* - I advocate for exporting injection points for dependencies and configuration. I also push people towards considering why something is or isn't exported; I simply don't believe the export-as-a-last resort philosophy for package we do not intent to be external API.
For example, I see many places where a sync.Mutex is embedded in a struct so (lazy) programmers can simply call lock on the self-reference without naming the lock. This exposes Lock and Unlock on the struct. I'd rarely (if ever) expect external lock management to be intended yet, we have it.
I also advocate for using _test packages to force authors to consider how the api surface will be used and what the capabilities are. If dependencies are structured properly, we can still get to very high code coverage. If we need tests in the main production package, so be it - but I don't believe that's the unilateral starting point.
As for internal packages, I already have a JIRA opened for that. I strongly believe we should restructure all of our packages and move the vast majority into an internal namespace.
yacovm (Thu, 09 Aug 2018 21:24:38 GMT):
> so (lazy) programmers can simply call lock on the self-reference without naming the lock
:raised_back_of_hand_tone1:
sykesm (Thu, 09 Aug 2018 21:46:40 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=aHDG8YXwm97ikLoaG) @yacovm yep. :)
yacovm (Thu, 09 Aug 2018 21:47:06 GMT):
Honestly I don't think all these things matter so much
sykesm (Thu, 09 Aug 2018 21:47:41 GMT):
Then I think you've just volunteered to do all of the peer cleanup
yacovm (Thu, 09 Aug 2018 21:48:01 GMT):
nah, i got better things to do
yacovm (Thu, 09 Aug 2018 21:48:13 GMT):
the peer needs cleanup, for sure
yacovm (Thu, 09 Aug 2018 21:48:28 GMT):
but we have bigger problems than just code style
sykesm (Thu, 09 Aug 2018 21:49:07 GMT):
I agree - but nothing exists in isolation. ttyl
guoger (Mon, 13 Aug 2018 08:38:34 GMT):
as I'm thinking about it a bit more, perhaps a method to notify `Chain` about leadership changes is need. Suppose following scenario:
- config seq is 1
- nodeA is leader. It has 2 txs in blockcutter: {normalMsg, configSeq: 1}, {normalMsg, configSeq: 1}
- nodeA loses leadership, nodeB becomes leader
- nodeB receives msg {configMsg, configSeq: 1}, it cuts a block and apply
- config seq now moves to 2
- nodeB loses leadership, nodeA becomes leader again
- nodeA cuts two _invalid_ messages into block due to timeout, and we have a problem.
(although, this sounds like an _uncle blocks_ problem to me, which may be solved by revalidating enqueued txs if a config block is being committed?)
guoger (Mon, 13 Aug 2018 08:38:43 GMT):
cc @C0rWin
C0rWin (Mon, 13 Aug 2018 09:29:02 GMT):
In such scenario I'd expect FSM to take care by not agreeing to consent on the block with smaller config sequence
C0rWin (Mon, 13 Aug 2018 09:30:59 GMT):
while handling Raft FSM step, we can check and reject propose of the block with old config sequence
guoger (Mon, 13 Aug 2018 09:32:01 GMT):
why `step`?
C0rWin (Mon, 13 Aug 2018 09:32:19 GMT):
this is where you handled transitions, no?
guoger (Mon, 13 Aug 2018 09:40:50 GMT):
why not `propose`?
guoger (Mon, 13 Aug 2018 09:41:12 GMT):
i don't think we should intercept `step` messages
kostas (Mon, 13 Aug 2018 14:27:39 GMT):
Good point @guoger.
kostas (Mon, 13 Aug 2018 14:27:39 GMT):
Good point @guoger.
kostas (Mon, 13 Aug 2018 14:27:39 GMT):
@guoger: Good point.
kostas (Mon, 13 Aug 2018 14:28:11 GMT):
I _think_ that such a signal is needed as well.
kostas (Mon, 13 Aug 2018 14:29:05 GMT):
@C0rWin: In the scenario that Jay describes, don't you (i.e. nodeA) run the risk of cutting a block with these stale 2 TXs even when node B has become the leader?
kostas (Mon, 13 Aug 2018 14:29:19 GMT):
Your batch timer will eventually expire.
kostas (Mon, 13 Aug 2018 14:44:24 GMT):
(A reminder to all that the package name is `etcdraft`, not `raft`. See: https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAAB62Du0s)
kostas (Mon, 13 Aug 2018 14:44:24 GMT):
(A reminder that the package name is `etcdraft`, not `raft`. See: https://docs.google.com/document/d/138Brlx2BiYJm5bzFk_B0csuEUKYdXXr7Za9V7C76dwo/edit?disco=AAAAB62Du0s)
C0rWin (Mon, 13 Aug 2018 15:24:40 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=hw97s66jDf8cA5jwH) @kostas this is why I have asked Jay to provide such signal, he just articulated my request here out of loud to induce dialog I guess
C0rWin (Mon, 13 Aug 2018 15:26:49 GMT):
Not sure why this a risk, though, no one will consent on this block, so eventually transactions will get lost. Personally I think that if will have such signal ex-leader could try to forward transactions to new one
guoger (Mon, 13 Aug 2018 15:30:33 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=36GaitmaADX4NcyxM) right, @C0rWin suggested to add such a signal between chain and fsm, and i wasn't sure about it. then I thought a bit more, and this scenario occurred to me as a solid reason.
guoger (Mon, 13 Aug 2018 15:33:01 GMT):
and i don't think it's a risk, because we guard `Propose` not to accept block as follower. (also, `DisableProposalForwarding` is enabled)
kostas (Mon, 13 Aug 2018 15:35:40 GMT):
Ah, I see - thanks for having this discussion in public then.
kostas (Mon, 13 Aug 2018 15:35:44 GMT):
Two quick comments:
kostas (Mon, 13 Aug 2018 15:35:49 GMT):
> Personally I think that if will have such signal ex-leader could try to forward transactions to new one
kostas (Mon, 13 Aug 2018 15:36:50 GMT):
Let's not do that for now. I'd rather we keep things simple, and not do anything that might be complicated for the BFT path.
guoger (Mon, 13 Aug 2018 15:37:07 GMT):
i agree
kostas (Mon, 13 Aug 2018 15:37:12 GMT):
> and i don't think it's a risk, because we guard `Propose` not to accept block as follower. (also, `DisableProposalForwarding` is enabled)
Expand on this one?
C0rWin (Mon, 13 Aug 2018 15:37:54 GMT):
Jay actually has a check which results in error if you as non leader will try to Propose block
kostas (Mon, 13 Aug 2018 15:38:46 GMT):
Ah, I remember now.
guoger (Mon, 13 Aug 2018 15:38:46 GMT):
> don't you (i.e. nodeA) run the risk of cutting a block with these stale 2 TXs even when node B has become the leader?
follower is not allowed to propose block to raft. And i also explicitly enable `DisableProposalForwarding` in raft.Config
kostas (Mon, 13 Aug 2018 15:39:08 GMT):
> And i also explicitly enable `DisableProposalForwarding` in raft.Config
I wasn't even aware this was a thing. Cool!
kostas (Mon, 13 Aug 2018 15:39:27 GMT):
Can someone capture this decision/impl. detail in either of your two JIRAs?
guoger (Mon, 13 Aug 2018 15:39:40 GMT):
I'll update my jira
kostas (Mon, 13 Aug 2018 15:39:44 GMT):
Add it as a comment. I know that my memory sucks.
kostas (Mon, 13 Aug 2018 15:39:49 GMT):
Cool, thanks Jay.
guoger (Mon, 13 Aug 2018 15:39:53 GMT):
but we need to decide on the form of signal
kostas (Mon, 13 Aug 2018 15:40:42 GMT):
Do either of you have proposal to begin with?
kostas (Mon, 13 Aug 2018 15:40:42 GMT):
Do either of you have a proposal to begin with?
guoger (Mon, 13 Aug 2018 15:41:07 GMT):
roughly, in `<-node.Ready()`, I'll compare `wasLeader` and current state, to detect leadership change
guoger (Mon, 13 Aug 2018 15:41:07 GMT):
roughly, in `<-node.Ready()`, I'll compare `wasLeader` and current state, to detect leadership change. (this is basically how I would implement it, just in case anybody has a better idea)
guoger (Mon, 13 Aug 2018 15:41:32 GMT):
and I would suggest a callback
guoger (Mon, 13 Aug 2018 15:41:32 GMT):
and I would suggest a callback
kostas (Mon, 13 Aug 2018 15:47:32 GMT):
Let's wait for Artem, and then we can chime in.
C0rWin (Mon, 13 Aug 2018 16:21:26 GMT):
Callback is implementation detail, honestly don’t really mind how this gonna be implemented as long as we agree this is must have API
kostas (Mon, 13 Aug 2018 17:48:17 GMT):
So @yacovm is right that the path that touches on updating the certificates on an existing replica/OSN is a bit hand-wavy.
kostas (Mon, 13 Aug 2018 17:48:42 GMT):
Remember that we settled on a high-level approach on this a month ago or so, which can be summarized as:
kostas (Mon, 13 Aug 2018 17:49:04 GMT):
This is a Fabric configuration update that should not trigger a Raft config update.
kostas (Mon, 13 Aug 2018 17:50:39 GMT):
So the question is: how do we actually process these transactions and keep track of the new mapping? i.e. it used to be that replica 2 has cert "foo" and now its cert changes to "bar". We need to ensure that the new mapping reads:
kostas (Mon, 13 Aug 2018 17:51:03 GMT):
```1: cert1
2: bar
3: cert3```
kostas (Mon, 13 Aug 2018 17:51:06 GMT):
Instead of:
kostas (Mon, 13 Aug 2018 17:51:20 GMT):
```1: cert1
3: cert3
4: bar```
kostas (Mon, 13 Aug 2018 17:51:34 GMT):
(@yacovm - Did I get the question right?)
yacovm (Mon, 13 Aug 2018 17:51:39 GMT):
yeah
kostas (Mon, 13 Aug 2018 17:51:42 GMT):
Cool.
kostas (Mon, 13 Aug 2018 17:51:54 GMT):
So here's what I'm thinking. And it's nothing exotic - standard process.
kostas (Mon, 13 Aug 2018 17:52:22 GMT):
The `Chain` implements a `Configure` method.
yacovm (Mon, 13 Aug 2018 17:52:44 GMT):
hold on... before you dive in the code
kostas (Mon, 13 Aug 2018 17:52:51 GMT):
Sure.
yacovm (Mon, 13 Aug 2018 17:53:00 GMT):
how do you distinguish?
yacovm (Mon, 13 Aug 2018 17:53:23 GMT):
i mean, conceptually
kostas (Mon, 13 Aug 2018 17:53:50 GMT):
I write logic that unpacks the `LastUpdate` of the proposed configuration and detects what's going on.
kostas (Mon, 13 Aug 2018 17:54:33 GMT):
I will need to do that anyway for that bit where we are supposed to reject any configuration update that increases/decreases the replica set by more than one node at a time. (Per the doc.)
kostas (Mon, 13 Aug 2018 17:55:55 GMT):
And on the leader side again, when they're about to create the config block and have it ordered, they will execute the same bit of processing logic, and update the config block's metadata (which carries the mapping) accordingly.
kostas (Mon, 13 Aug 2018 17:56:39 GMT):
In fact, now that I think of it, this bit of logic should only be executed on the leader before they ask to have the config block ordered.
kostas (Mon, 13 Aug 2018 17:56:44 GMT):
We don't need it on ingress.
kostas (Mon, 13 Aug 2018 17:56:59 GMT):
The only custom piece of config processing logic on ingress should be:
kostas (Mon, 13 Aug 2018 17:57:03 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=J25FrcpgCjgpey9Bu
kostas (Mon, 13 Aug 2018 17:57:47 GMT):
So, any comm layer changes will have to take effect once a new config block (with its associated metadata) is written to the ledger.
yacovm (Mon, 13 Aug 2018 17:58:10 GMT):
I still don't understand how you distinguish the 2
kostas (Mon, 13 Aug 2018 17:58:44 GMT):
Q1: Do I, the code author, have access to the most recent block metadata?
yacovm (Mon, 13 Aug 2018 17:59:09 GMT):
lets say you have access to any data in the world and you are computationally un-bounded
yacovm (Mon, 13 Aug 2018 17:59:24 GMT):
how do you understand the config update's intent?
kostas (Mon, 13 Aug 2018 17:59:52 GMT):
I read the `LastUpdate` and extract its `Writeset`?
kostas (Mon, 13 Aug 2018 18:00:13 GMT):
And if what it wishes to update is the Consenter set, I go over each item and keep track of what's being modified?
yacovm (Mon, 13 Aug 2018 18:00:17 GMT):
yeah, you know the previous mapping
yacovm (Mon, 13 Aug 2018 18:00:24 GMT):
you know the previous mapping of IDs to certificates
yacovm (Mon, 13 Aug 2018 18:00:33 GMT):
but on the next config update you don't have the mapping
yacovm (Mon, 13 Aug 2018 18:00:56 GMT):
and i understand that you can backward map the certs to the ID
yacovm (Mon, 13 Aug 2018 18:01:06 GMT):
but how do you know if the ID changes or you should add a new ID?
yacovm (Mon, 13 Aug 2018 18:01:06 GMT):
but how do you know if the certs changes or you should add a new ID?
kostas (Mon, 13 Aug 2018 18:01:37 GMT):
(I may have to cut off temporarily shortly, but I'll be back.)
kostas (Mon, 13 Aug 2018 18:01:38 GMT):
So --
kostas (Mon, 13 Aug 2018 18:01:47 GMT):
When I know the metadata I know:
kostas (Mon, 13 Aug 2018 18:02:15 GMT):
`Raft node ID - OSN.host+port - OSN.certs`
C0rWin (Mon, 13 Aug 2018 18:02:19 GMT):
Cannot we just add node id to solve your question @yacovm ?
yacovm (Mon, 13 Aug 2018 18:02:29 GMT):
that's what I am recommending Artem :)
kostas (Mon, 13 Aug 2018 18:03:09 GMT):
Folks, I've got about 5 pages in the document explaining why the node ID is a no go.
C0rWin (Mon, 13 Aug 2018 18:03:11 GMT):
Oh, I see. Having node id will allow to clearly and very easy distinguish
yacovm (Mon, 13 Aug 2018 18:03:45 GMT):
Kostas - I just want to understand how we distinguish... that's all
kostas (Mon, 13 Aug 2018 18:03:54 GMT):
And I'm in the process of explaining it.
kostas (Mon, 13 Aug 2018 18:04:10 GMT):
> When I know the metadata I know: `Raft node ID - OSN.host+port - OSN.certs`
kostas (Mon, 13 Aug 2018 18:04:14 GMT):
Do you agree with this?
yacovm (Mon, 13 Aug 2018 18:04:17 GMT):
yeah
kostas (Mon, 13 Aug 2018 18:04:35 GMT):
Cool. And the writeset of the Fabric config update will include:
kostas (Mon, 13 Aug 2018 18:04:46 GMT):
`OSN.host+port - new.OSN.cert`
kostas (Mon, 13 Aug 2018 18:05:09 GMT):
So I can then join these two pieces and do the updating?
kostas (Mon, 13 Aug 2018 18:05:29 GMT):
`OSN.host+port` is common between the two.
yacovm (Mon, 13 Aug 2018 18:05:39 GMT):
you're saying you are using the OSN.host:port to map to the ID
kostas (Mon, 13 Aug 2018 18:06:06 GMT):
During this kind update, yes.
kostas (Mon, 13 Aug 2018 18:06:14 GMT):
Do you see any flaws to this approach?
yacovm (Mon, 13 Aug 2018 18:07:30 GMT):
that's what I was asking you
yacovm (Mon, 13 Aug 2018 18:07:39 GMT):
and I'm not sure we can say it is
kostas (Mon, 13 Aug 2018 18:07:51 GMT):
I see your message in the private chat now.
kostas (Mon, 13 Aug 2018 18:07:58 GMT):
You had indeed suggested this but I missed it.
yacovm (Mon, 13 Aug 2018 18:08:12 GMT):
it might be that we need to change the host and because of that - we need to also change the server cert because we would need to change the DNS/IP SAN
kostas (Mon, 13 Aug 2018 18:08:15 GMT):
> and I'm not sure we can say it is
We can say it is... flawed?
kostas (Mon, 13 Aug 2018 18:08:39 GMT):
We are overthinking this.
kostas (Mon, 13 Aug 2018 18:08:51 GMT):
If the host needs to change, drop the node and add it as a new one.
kostas (Mon, 13 Aug 2018 18:08:56 GMT):
Optimize for the common case.
yacovm (Mon, 13 Aug 2018 18:09:22 GMT):
What will raft do in such a case, btw?
kostas (Mon, 13 Aug 2018 18:09:36 GMT):
Meaning?
yacovm (Mon, 13 Aug 2018 18:09:45 GMT):
you have a new node that has a valid log
yacovm (Mon, 13 Aug 2018 18:09:53 GMT):
will Raft handle that gracefully ?
yacovm (Mon, 13 Aug 2018 18:10:08 GMT):
surely, the other nodes will think at least at first, that it's a new node
kostas (Mon, 13 Aug 2018 18:10:08 GMT):
Yes.
kostas (Mon, 13 Aug 2018 18:10:21 GMT):
It will find its place and sync up as needed, no issues.
yacovm (Mon, 13 Aug 2018 18:10:44 GMT):
so if we are over-optimizing this, I suggest we don't do the endpoint (host:port) mapping idea
yacovm (Mon, 13 Aug 2018 18:10:53 GMT):
and go full new node each time
yacovm (Mon, 13 Aug 2018 18:11:01 GMT):
less code paths to check in integration
yacovm (Mon, 13 Aug 2018 18:11:11 GMT):
and if you say raft handles this well - then we need to always use it
kostas (Mon, 13 Aug 2018 18:11:11 GMT):
This is actually totally fine with me as well.
kostas (Mon, 13 Aug 2018 18:11:36 GMT):
Depends on what our assumption is regarding TLS certs and how often they need to be rotated and changed.
yacovm (Mon, 13 Aug 2018 18:11:53 GMT):
I'd say once in every 3 months is good
kostas (Mon, 13 Aug 2018 18:11:58 GMT):
I believe this is how this whole convo started in the Google Doc as well, following Gari's observation.
yacovm (Mon, 13 Aug 2018 18:13:15 GMT):
hey I'm fine with everything we do as long as we are convinced it should work...
kostas (Mon, 13 Aug 2018 18:14:01 GMT):
Heh. We are actually not convinced it works until the integration test passes.
kostas (Mon, 13 Aug 2018 18:14:27 GMT):
I mean, we do our best to study the paths and account for side-effects, but we'll certainly miss stuff.
kostas (Tue, 14 Aug 2018 14:53:21 GMT):
Have we captured the `OnStep/OnSubmit` contract in JIRA or in the doc?
yacovm (Tue, 14 Aug 2018 14:55:26 GMT):
It's a smart contract
yacovm (Tue, 14 Aug 2018 14:55:56 GMT):
seriously - it's just an API between the communication and the consumers of it
yacovm (Tue, 14 Aug 2018 14:55:56 GMT):
it's just an API between the communication and the consumers of it
kostas (Tue, 14 Aug 2018 15:21:30 GMT):
Where is this contract documented?
yacovm (Tue, 14 Aug 2018 15:21:59 GMT):
no idea....
yacovm (Tue, 14 Aug 2018 15:22:13 GMT):
@kostas - i want to push the communication today - can i just use the story JIRA?
yacovm (Tue, 14 Aug 2018 15:22:22 GMT):
I want to push several commits with the same JIRA number
kostas (Tue, 14 Aug 2018 15:25:12 GMT):
I'd create a sub-task under FAB-11161 capturing the specifics of the work (for you it's most likely the description you wrote in FAB-11161 - you can copy and paste it if that's the case) and submit it against that sub-task.
kostas (Tue, 14 Aug 2018 15:25:50 GMT):
I want FAB-11161 to be the parent story for Week 6-8: https://docs.google.com/spreadsheets/d/1R02-xjdl4hNw90kN4NZ1kX0f4QoIYn51_1UHlwy-GQs/edit#gid=2125670104
kostas (Tue, 14 Aug 2018 15:26:10 GMT):
So we may wish to add more sub-tasks to it later on.
yacovm (Tue, 14 Aug 2018 15:26:24 GMT):
ok so i will open a new one
yacovm (Tue, 14 Aug 2018 15:26:30 GMT):
and submit 2 change sets for the same JIRA
yacovm (Tue, 14 Aug 2018 15:26:34 GMT):
makes sense to you?
kostas (Tue, 14 Aug 2018 15:26:57 GMT):
Sure.
yacovm (Tue, 14 Aug 2018 15:44:36 GMT):
oook
yacovm (Tue, 14 Aug 2018 15:44:39 GMT):
are you ready?
yacovm (Tue, 14 Aug 2018 15:45:04 GMT):
https://gerrit.hyperledger.org/r/#/c/25533/
https://gerrit.hyperledger.org/r/#/c/25535/
tadaaa
yacovm (Tue, 14 Aug 2018 15:45:24 GMT):
1) 100% code coverage, runs in ~ 2 seconds on average
2) no data races (I checked)
kostas (Tue, 14 Aug 2018 15:46:33 GMT):
Got it, thanks Yacov. Will review.
yacovm (Tue, 14 Aug 2018 15:54:36 GMT):
sure, if something isn't clear feel free to ask, but i tried to put lots of documentation in the code to make it clear
kostas (Tue, 14 Aug 2018 16:53:08 GMT):
I'm writing down all the stories with acceptance criteria so that we can have some backlog.
kostas (Tue, 14 Aug 2018 16:53:17 GMT):
Here's an issue I think I've bumped into.
kostas (Tue, 14 Aug 2018 16:53:53 GMT):
It has to do with how we process configuration update transactions that modify the consenter set (by just one node, i.e. valid configuration updates).
kostas (Tue, 14 Aug 2018 16:54:27 GMT):
As a spoiler alert, I think it is an invalid edge case, but let's make sure.
kostas (Tue, 14 Aug 2018 16:54:45 GMT):
You have only 1 ordering service node. (Which is below the minimum 3.)
kostas (Tue, 14 Aug 2018 17:04:13 GMT):
Ah, nevermind -- as I was writing out the scenario, I figured out that we're covered against it.
guoger (Tue, 14 Aug 2018 17:19:32 GMT):
@kostas updated https://jira.hyperledger.org/browse/FAB-11162 to document `FSM` APIs
guoger (Tue, 14 Aug 2018 17:21:21 GMT):
tomorrow I will 1) implement `OnLeaderChange` callback and document it, 2) rename pkg to etcdraft, 3) address remaining comments on that CR, namely storage engine
kostas (Tue, 14 Aug 2018 17:37:35 GMT):
Jay: thanks! For the storage engine comment, what I'm driving at is that "expose (almost) everything" mentality that I'd like us to adopt for this package. Matt's advice comes down to "exposing the injection points", but I'd take it a step further for now and say "expose almost everything" because we can't properly tell the injection points for now. (The storage engine comment is a perfect example of this.)
guoger (Wed, 15 Aug 2018 09:16:43 GMT):
@C0rWin I pushed a revision here https://gerrit.hyperledger.org/r/c/24919/12 to add `IsLeader() bool` method. I didn't do callback because:
- i'm not sure if it's blocking
- if there's an error, `fsm` is not able to handle it.
if you check leadership when you are about to cut a block and act accordingly, it should be enough?
Also, jira is updated
C0rWin (Wed, 15 Aug 2018 09:18:20 GMT):
Thanks, no I am not blocked
guoger (Wed, 15 Aug 2018 09:23:51 GMT):
oh wait, I just realized this doesn't solve this problem https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=zv3LW5nDNCDXqxwvu
you need to be _immediately_ notified upon leadership change to purge blockcutter, right?
kostas (Wed, 15 Aug 2018 10:35:58 GMT):
Correct, the blockcutter needs to be purged right a way. You’re looking at something like LeaderChanged and it returns an empty struct channel.
kostas (Wed, 15 Aug 2018 10:35:58 GMT):
Correct, the blockcutter needs to be purged right away. You’re looking at something like LeaderChanged and it returns an empty struct channel.
kostas (Wed, 15 Aug 2018 10:35:58 GMT):
Correct, the blockcutter needs to be purged right away. You’re probably looking at something like `LeaderChanged() <- chan struct{}`.
kostas (Wed, 15 Aug 2018 12:33:22 GMT):
FWIW: https://github.com/coreos/etcd/issues/9965
kostas (Wed, 15 Aug 2018 14:35:18 GMT):
@yacovm: If one were to review your CRs, what would you suggest they start with? Is there maybe a specific test that will make everything sink in? A 3-line high-level overview of how your structs interact with each other?
yacovm (Wed, 15 Aug 2018 14:35:33 GMT):
the basic tet ;)
yacovm (Wed, 15 Aug 2018 14:35:33 GMT):
the basic test ;)
yacovm (Wed, 15 Aug 2018 14:35:36 GMT):
*test
kostas (Wed, 15 Aug 2018 14:36:15 GMT):
I figured (https://gerrit.hyperledger.org/r/c/25533/4/orderer/common/cluster/comm_test.go#222), but then again, you never know, and I guess I'm fishing more for the second part of the question.
yacovm (Wed, 15 Aug 2018 14:36:28 GMT):
hold on please, in a call
kostas (Wed, 15 Aug 2018 14:36:32 GMT):
No rush.
yacovm (Wed, 15 Aug 2018 14:46:21 GMT):
aah so w.r.t how structs interact with each other - the `Comm` struct is the communication implementation https://gerrit.hyperledger.org/r/#/c/25533/4/orderer/common/cluster/comm.go@80
and it uses a `ConnectionStore` to store the connections https://gerrit.hyperledger.org/r/#/c/25533/4/orderer/common/cluster/connections.go@39 mapped by TLS certificates
yacovm (Wed, 15 Aug 2018 14:46:34 GMT):
that's basically the first CR
kostas (Wed, 15 Aug 2018 14:47:03 GMT):
Perfect, that should get me going.
yacovm (Wed, 15 Aug 2018 14:47:04 GMT):
now for the second - the `service.go` is the implementation of the gRPC service itself but it just dispatches stuff into `Comm`
yacovm (Wed, 15 Aug 2018 14:47:16 GMT):
and a corresponding "client side" is the `rpc.go`
kostas (Wed, 15 Aug 2018 14:47:18 GMT):
Hold off on the second for now. (Or post, and be ready for Qs later.)
yacovm (Wed, 15 Aug 2018 14:47:19 GMT):
they both use `Comm`
kostas (Thu, 16 Aug 2018 16:10:46 GMT):
Do we have an educated take as to whether we should be using `require` versus `assert`?
kostas (Thu, 16 Aug 2018 16:24:46 GMT):
Do we have an educated take as to whether our `assert` statements in tests should be converted to `require`? We're asserting, but as I think about it, we should be requiring.
yacovm (Thu, 16 Aug 2018 16:25:05 GMT):
why?
kostas (Thu, 16 Aug 2018 16:26:09 GMT):
Fail fast?
yacovm (Thu, 16 Aug 2018 16:28:03 GMT):
I dont fail
yacovm (Thu, 16 Aug 2018 16:28:28 GMT):
If the test fails
yacovm (Thu, 16 Aug 2018 16:28:35 GMT):
IT is brittle
yacovm (Thu, 16 Aug 2018 16:28:35 GMT):
It is brittle
yacovm (Thu, 16 Aug 2018 16:28:50 GMT):
So the real problem is in the test
yacovm (Thu, 16 Aug 2018 16:29:03 GMT):
No in the command you use- require/assert
yacovm (Thu, 16 Aug 2018 16:29:59 GMT):
And if we need to use require instead of asset to figure out the problem then we have a bigger problem
kostas (Thu, 16 Aug 2018 16:30:14 GMT):
Not sure I follow the reasoning.
yacovm (Thu, 16 Aug 2018 16:30:35 GMT):
I'm saying it doesnt matter
yacovm (Thu, 16 Aug 2018 16:31:05 GMT):
Neither should fail
kostas (Thu, 16 Aug 2018 16:55:37 GMT):
> Neither should fail
Who argued for the opposite?
kostas (Thu, 16 Aug 2018 16:55:55 GMT):
The whole point is whether we should be failing fast. And I think we should.
kostas (Thu, 16 Aug 2018 16:56:21 GMT):
See first paragraph here: https://npf.io/2017/08/lies/
yacovm (Thu, 16 Aug 2018 17:38:12 GMT):
If you have parallel code that has asserts in goroutines, then having them all logged in the test failure, might give you more information why the test failed than if you had require and you'd fail fast
yacovm (Thu, 16 Aug 2018 17:38:53 GMT):
if you want me to change the asserts to requires in my code, please say so in the code review
yacovm (Thu, 16 Aug 2018 17:42:10 GMT):
but keep in mind it would take time, and once you waste time you never reclaim it
C0rWin (Thu, 16 Aug 2018 18:02:22 GMT):
> We're asserting, but as I think about it, we should be requiring.
does "fail fast" the only pros of `require` vs `assert`?
guoger (Mon, 20 Aug 2018 11:21:43 GMT):
should we actually wait for a block to be consented before proposing next block? per https://gerrit.hyperledger.org/r/c/25587/5/orderer/consensus/raft/chain.go#201
cc @C0rWin @kostas
guoger (Mon, 20 Aug 2018 11:23:56 GMT):
i thought the order is persisted: _ on raft leader, if `propose(A)` is called before `propose(B)`, then A is guaranteed to be ready before B_
kostas (Mon, 20 Aug 2018 11:25:51 GMT):
Right now, we _have to_ do that. Because `CreateNextBlock` and `WriteNextBlock` are tightly coupled.
kostas (Mon, 20 Aug 2018 11:26:11 GMT):
See footnote 13 on the document and Week 11 on the spreadsheet (when we'll address this).
guoger (Mon, 20 Aug 2018 11:26:14 GMT):
oh right, I forgot about that
guoger (Mon, 20 Aug 2018 11:26:32 GMT):
thx
kostas (Mon, 20 Aug 2018 11:26:47 GMT):
Right, so _today_ we are unable to calculate the hash of B unless A has been written on our ledger.
bdjidi (Tue, 21 Aug 2018 22:48:34 GMT):
Has joined the channel.
kostas (Wed, 22 Aug 2018 12:19:58 GMT):
Jay's experimenting with a joint chain+FSM approach in his latest patchset: https://gerrit.hyperledger.org/r/c/24919/
kostas (Wed, 22 Aug 2018 12:20:39 GMT):
This is a divergence from the original design but an inexpensive exploration, esp. given that Artem's CR is a WIP.
kostas (Wed, 22 Aug 2018 12:20:59 GMT):
We review and if it makes sense, we roll with it. If not, we revert back. Code is malleable.
yacovm (Sun, 26 Aug 2018 20:02:12 GMT):
Please take a look if the approach makes sense and if I'm in the right direction.
https://gerrit.hyperledger.org/r/#/c/25891/1/orderer/consensus/etcdraft/dispatcher.go
If yes - then I'll add tests
guoger (Mon, 27 Aug 2018 14:39:26 GMT):
@kostas added sender ID per your request. Also, I think you accidentally updated my CR when you uploaded yours :P
kostas (Mon, 27 Aug 2018 14:40:45 GMT):
(Probably, Yacov right?)
guoger (Mon, 27 Aug 2018 14:42:30 GMT):
oh, @yacovm , wrong person :P
yacovm (Mon, 27 Aug 2018 14:47:17 GMT):
I just rebased your CR though
yacovm (Mon, 27 Aug 2018 14:47:23 GMT):
Did not touch the code
yacovm (Mon, 27 Aug 2018 14:47:47 GMT):
At least i think so
guoger (Mon, 27 Aug 2018 14:49:17 GMT):
ah, ok. I didn't see some newly added files back then so I thought it may get lost while being updated. Anyways, just rebased and added `sender`
kostas (Mon, 27 Aug 2018 15:34:00 GMT):
Looking at the dispatcher CR. Let's expand a bit on how we expect it to be used in practice. We need a singleton that implements the ReceiverGetter interface in that CR. What is this singleton tasked with other than implementing that interface?
kostas (Mon, 27 Aug 2018 15:34:32 GMT):
For instance, this will most likely be the singleton binding the grpcServer with the clusterServer, no?
yacovm (Mon, 27 Aug 2018 16:58:03 GMT):
No....
yacovm (Mon, 27 Aug 2018 16:58:20 GMT):
We just need to somehow connect it to the regostrar
yacovm (Mon, 27 Aug 2018 16:58:25 GMT):
*regostrar
yacovm (Mon, 27 Aug 2018 16:58:38 GMT):
*registrar
kostas (Mon, 27 Aug 2018 17:07:04 GMT):
Expand on this please.
yacovm (Mon, 27 Aug 2018 17:07:31 GMT):
it's easy...
kostas (Mon, 27 Aug 2018 17:07:35 GMT):
I am dumb.
yacovm (Mon, 27 Aug 2018 17:07:48 GMT):
we need someone to give you the instances of the consenters/chains
yacovm (Mon, 27 Aug 2018 17:07:49 GMT):
right?
yacovm (Mon, 27 Aug 2018 17:07:53 GMT):
that's what that thing does
yacovm (Mon, 27 Aug 2018 17:08:02 GMT):
the registrar currently maps this, no?
yacovm (Mon, 27 Aug 2018 17:08:26 GMT):
it holds the consensus specific pointer to the consenter
yacovm (Mon, 27 Aug 2018 17:08:35 GMT):
so we need somehow, i think -to connect from that point
yacovm (Mon, 27 Aug 2018 17:08:38 GMT):
to where we are
kostas (Mon, 27 Aug 2018 17:09:12 GMT):
Alright, hear me out for a bit.
kostas (Mon, 27 Aug 2018 17:09:31 GMT):
We need to bind the gRPC server to the clusterServer.
yacovm (Mon, 27 Aug 2018 17:09:33 GMT):
sure
yacovm (Mon, 27 Aug 2018 17:09:49 GMT):
to the *Service* ;)
yacovm (Mon, 27 Aug 2018 17:09:53 GMT):
`service.go`
kostas (Mon, 27 Aug 2018 17:10:48 GMT):
https://github.com/hyperledger/fabric/blob/release-1.2/orderer/common/server/main.go#L109
kostas (Mon, 27 Aug 2018 17:11:22 GMT):
Likewise, we'll be looking at an `op.RegisterClusterServer(grpcServer.Server(), clusterServerImplGoesHere)`, correct?
yacovm (Mon, 27 Aug 2018 17:12:00 GMT):
ok
kostas (Mon, 27 Aug 2018 17:12:49 GMT):
And presumably this call will reside in a `[consenters"etcdraft"] = etcdraft.New(...)` line that will live here: https://github.com/hyperledger/fabric/blob/release-1.2/orderer/common/server/main.go#L258 before we pass on the `consenters` map to the registrar constructor.
kostas (Mon, 27 Aug 2018 17:13:30 GMT):
Now, we need to do the binding _once_.
yacovm (Mon, 27 Aug 2018 17:14:32 GMT):
Kostas can we talk about this tomorrow in a screen sharing session?
kostas (Mon, 27 Aug 2018 17:14:45 GMT):
Of course.
yacovm (Mon, 27 Aug 2018 17:14:50 GMT):
cool thanks
yacovm (Mon, 27 Aug 2018 20:08:58 GMT):
@guoger
https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/3978/console
yacovm (Mon, 27 Aug 2018 20:09:11 GMT):
```
15:12:25 unit-tests_1 | ••••••••••panic: runtime error: invalid memory address or nil pointer dereference
15:12:25 unit-tests_1 | [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xdd259f]
15:12:25 unit-tests_1 |
15:12:25 unit-tests_1 | goroutine 214 [running]:
15:12:25 unit-tests_1 | github.com/hyperledger/fabric/orderer/consensus/etcdraft_test.glob..func1.2.3.1(0x2, 0xc420312d20, 0x4, 0xc42000dde0, 0x2)
15:12:25 unit-tests_1 | /opt/gopath/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain_test.go:192 +0x4f
15:12:25 unit-tests_1 | github.com/hyperledger/fabric/orderer/consensus/etcdraft/etcdraftfakes.(*FakeTransport).Step(0xc420212a90, 0x2, 0xc420312d20, 0x1c, 0x1c, 0x2)
15:12:25 unit-tests_1 | /opt/gopath/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/etcdraftfakes/fake_transport.go:52 +0x273
15:12:25 unit-tests_1 | github.com/hyperledger/fabric/orderer/consensus/etcdraft.(*Chain).send(0xc42001ccc0, 0xc4202d4000, 0x1, 0x1)
15:12:25 unit-tests_1 | /opt/gopath/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain.go:417 +0x1f7
15:12:25 unit-tests_1 | github.com/hyperledger/fabric/orderer/consensus/etcdraft.(*Chain).serveRaft(0xc42001ccc0)
15:12:25 unit-tests_1 | /opt/gopath/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain.go:326 +0x2ad
15:12:25 unit-tests_1 | created by github.com/hyperledger/fabric/orderer/consensus/etcdraft.(*Chain).Start
15:12:25 unit-tests_1 | /opt/gopath/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain.go:141 +0xdf
```
yacovm (Mon, 27 Aug 2018 21:07:01 GMT):
@guoger I rebased your https://gerrit.hyperledger.org/r/#/c/24919/ on top of my https://gerrit.hyperledger.org/r/#/c/25891/
yacovm (Mon, 27 Aug 2018 21:07:05 GMT):
hope you don't mind :)
yacovm (Mon, 27 Aug 2018 21:07:47 GMT):
https://gerrit.hyperledger.org/r/#/c/25479/ is conflicted though, and can't be rebased without non trivial conflict resolution
yacovm (Tue, 28 Aug 2018 16:07:58 GMT):
@kostas please merge https://gerrit.hyperledger.org/r/#/c/25891/ :)
adarshsaraf123 (Wed, 29 Aug 2018 11:58:21 GMT):
Has anybody tried using `delve` with either `orderer`, `configtxlator`, `configtxgen`? I keep getting this error:
`could not launch process: decoding dwarf section info at offset 0x0: too short`
even with `dlv debug`.
kostas (Wed, 29 Aug 2018 17:29:47 GMT):
@adarshsaraf123: I have and have bumped into the exact same issue.
kostas (Wed, 29 Aug 2018 17:29:59 GMT):
Unfortunately, I never bothered to investigate it further.
adarshsaraf123 (Wed, 29 Aug 2018 17:31:29 GMT):
@kostas Thanks for the confirmation. I have tried long enough and have given up :sweat_smile:
yacovm (Thu, 30 Aug 2018 08:43:28 GMT):
@guoger
yacovm (Thu, 30 Aug 2018 08:43:36 GMT):
what is this clocking thing in your CR
yacovm (Thu, 30 Aug 2018 08:43:43 GMT):
and what is its purpose in life?
guoger (Thu, 30 Aug 2018 08:44:00 GMT):
clocking bool?
yacovm (Thu, 30 Aug 2018 08:44:03 GMT):
yeah
guoger (Thu, 30 Aug 2018 08:45:09 GMT):
indicate if timer is alive
yacovm (Thu, 30 Aug 2018 08:45:25 GMT):
alive?
guoger (Thu, 30 Aug 2018 08:46:04 GMT):
if it's running
guoger (Thu, 30 Aug 2018 08:46:23 GMT):
(not sure how to describe it in proper english :P)
yacovm (Thu, 30 Aug 2018 08:46:32 GMT):
so basically...
yacovm (Thu, 30 Aug 2018 08:46:47 GMT):
`serveRequest` both tells blocks to be cut
yacovm (Thu, 30 Aug 2018 08:46:54 GMT):
and also orders into the block cutter?
guoger (Thu, 30 Aug 2018 08:48:13 GMT):
it's essentially `main()` in solo/consensus.go
yacovm (Thu, 30 Aug 2018 08:49:21 GMT):
I admit i don't understand something fundamental
yacovm (Thu, 30 Aug 2018 08:49:53 GMT):
oh, nevermind. got it
guoger (Thu, 30 Aug 2018 08:55:41 GMT):
how exactly do we determine node IDs? I submitted [this CR](https://gerrit.hyperledger.org/r/c/25964/) to implement following statement in design doc page 16
> During this iteration, each consenter also compares its own TLS certificates with the ones in the slice in order to detect its own node ID. Both its own ID, and the list of IDs it collected during the slice iteration, should be passed on to the Raft FSM constructor. (See Section C.I.3 "The Raft FSM".)
I'm not exactly sure where to get _own TLS cert_?
guoger (Thu, 30 Aug 2018 08:56:41 GMT):
appreciate some enlightenments...
adarshsaraf123 (Thu, 30 Aug 2018 09:17:44 GMT):
I guess that should be a part of localconfig.EtcdRaft isnt it? Each orderer node should be able to locally specify the same in orderer.yaml
adarshsaraf123 (Thu, 30 Aug 2018 09:17:44 GMT):
I guess that should be a part of localconfig.EtcdRaft, isnt it? Each orderer node should be able to locally specify the same in orderer.yaml
yacovm (Thu, 30 Aug 2018 09:35:16 GMT):
your own TLS certificate is from the orderer global config no?
yacovm (Thu, 30 Aug 2018 09:49:55 GMT):
```
batches, _ := c.support.BlockCutter().Ordered(msg.Content)
if len(batches) == 0 {
if !clocking {
clocking = true
timer.Reset(c.support.SharedConfig().BatchTimeout())
}
continue
}
```
@guoger - so if a block hasn't been cut yet, you reset the timer... can you explain why?
guoger (Thu, 30 Aug 2018 09:51:02 GMT):
*if* a block is not cut *and* timer is not started, *then* we start the timer
yacovm (Thu, 30 Aug 2018 09:51:24 GMT):
oh
guoger (Thu, 30 Aug 2018 09:52:09 GMT):
> your own TLS certificate is from the orderer global config no?
are we ever going to rotate cert?
yacovm (Thu, 30 Aug 2018 09:52:41 GMT):
yeah we can
yacovm (Thu, 30 Aug 2018 09:53:00 GMT):
but we need a gRPC admin service for that
yacovm (Thu, 30 Aug 2018 09:53:02 GMT):
which we don't have
kostas (Thu, 30 Aug 2018 16:52:06 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=jsvfM5Gmn9eCS6JEM
kostas (Thu, 30 Aug 2018 16:52:21 GMT):
@guoger: ^^ That is correct. Also have a look at this comment: https://gerrit.hyperledger.org/r/c/25235/4#message-3869ab34_48116562
kostas (Thu, 30 Aug 2018 16:59:16 GMT):
Actually, what on earth am I talking about.
kostas (Thu, 30 Aug 2018 16:59:33 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=Q5BA3agm8mym9csGS
kostas (Thu, 30 Aug 2018 16:59:39 GMT):
^^ This is correct.
kostas (Thu, 30 Aug 2018 17:00:03 GMT):
The Consenter knows what its own certs are because it loads them from the localconfig.
kostas (Thu, 30 Aug 2018 17:00:19 GMT):
And it iterarates over the global config to find them in that slice.
kostas (Thu, 30 Aug 2018 17:00:57 GMT):
From a very-very early draft I was working on way back:
kostas (Thu, 30 Aug 2018 17:01:02 GMT):
draggingScreenshot.png
kostas (Thu, 30 Aug 2018 17:01:27 GMT):
(Back from before we realized that we need both clients _and_ server TLS certs.)
yacovm (Thu, 30 Aug 2018 19:08:14 GMT):
@kostas obviously you need to know your own TLS cert to know what to look in the etcdraft configuration. there is no going around this
guoger (Fri, 31 Aug 2018 02:30:10 GMT):
@kostas can you take a look at [this](https://gerrit.hyperledger.org/r/c/25964/) to see if it's on right track? thx
adarshsaraf123 (Fri, 31 Aug 2018 21:18:36 GMT):
@kostas We will need to assert that each of the raft consenter nodes are listening on the host, port combination as specified in its `Consenter` info in the `metadata`. Am I right?
guoger (Tue, 04 Sep 2018 13:41:13 GMT):
from @yacovm
> we need to add the stuff you put in the config
yes, and @adarshsaraf123 has a patch for this
yacovm (Tue, 04 Sep 2018 13:41:40 GMT):
oh
yacovm (Tue, 04 Sep 2018 13:41:43 GMT):
is it already there?
adarshsaraf123 (Tue, 04 Sep 2018 13:42:09 GMT):
Pushing the patch now..
yacovm (Tue, 04 Sep 2018 13:42:24 GMT):
oh cool thanks
adarshsaraf123 (Tue, 04 Sep 2018 13:57:05 GMT):
https://gerrit.hyperledger.org/r/#/c/26074/
kostas (Tue, 04 Sep 2018 14:28:12 GMT):
@guoger: Sorry for the radio silence, I was sick most of last week.
kostas (Tue, 04 Sep 2018 14:28:16 GMT):
Will look now.
kostas (Tue, 04 Sep 2018 14:28:26 GMT):
I'll check the CRs from Adarsh and Yacov today as well.
kostas (Tue, 04 Sep 2018 14:54:45 GMT):
That `timer` API is killing me.
kostas (Tue, 04 Sep 2018 14:54:49 GMT):
> To prevent a timer created with NewTimer from firing after a call to Stop, check the return value and drain the channel.
kostas (Tue, 04 Sep 2018 14:54:56 GMT):
```if !t.Stop() {
<-t.C
}```
kostas (Tue, 04 Sep 2018 14:55:17 GMT):
We enter that branch if `t.Stop()` returns false.
kostas (Tue, 04 Sep 2018 14:55:38 GMT):
According to the Godoc, `t.Stop()` returns false is the timer "has already expired or been stopped."
kostas (Tue, 04 Sep 2018 14:56:17 GMT):
Also from the Godoc: "If a program has already received a value from t.C, the timer is known to have expired..."
kostas (Tue, 04 Sep 2018 14:56:58 GMT):
So, if you're hitting that branch when the timer has expired (i.e. someone has already received the value from `t.C`, aren't you going to be stuck in `<-t.C`?
kostas (Tue, 04 Sep 2018 14:57:07 GMT):
Someone enlighten me. I'm obviously missing something.
kostas (Tue, 04 Sep 2018 14:59:40 GMT):
Ah, the GoDoc does note: "This cannot be done concurrent to other receives from the Timer's channel."
kostas (Tue, 04 Sep 2018 15:02:01 GMT):
Still confused however. Wouldn't I want to drain `t.C` if t.Stop() returns true as well?
guoger (Tue, 04 Sep 2018 15:12:08 GMT):
if `t.Stop()` returns true, then timer is guaranteed to be explicitly stopped, no signal sent on channel, so you don't need to drain it
guoger (Tue, 04 Sep 2018 15:12:28 GMT):
maybe you can take a look at https://gerrit.hyperledger.org/r/c/25975, which wraps timer to hide these details
kostas (Tue, 04 Sep 2018 15:12:46 GMT):
Just reviewed that one and still had that question :grin:
kostas (Tue, 04 Sep 2018 15:13:00 GMT):
I like the work on that CR though - chain.go reads much cleaner.
guoger (Tue, 04 Sep 2018 15:13:11 GMT):
(thanks to @yacovm 's suggestion)
kostas (Tue, 04 Sep 2018 15:13:33 GMT):
Still confused damn it.
kostas (Tue, 04 Sep 2018 15:13:44 GMT):
So:
kostas (Tue, 04 Sep 2018 15:14:05 GMT):
In our case `t.Stop` will return false only if the timer has already expired.
kostas (Tue, 04 Sep 2018 15:18:45 GMT):
(Added a follow-up comment in that CR.)
guoger (Tue, 04 Sep 2018 15:20:59 GMT):
not exactly. i.e.
```
batches, _ := c.support.BlockCutter().Ordered(msg.Content)
if len(batches) == 0 {
timer.start(c.support.SharedConfig().BatchTimeout())
continue
}
timer.stop()
```
and suppose following scenario:
chain receives an envelope exceeding batch size, and this is the first envelope (timer was not started previously).
In this case, `timer.stop()` would stop a stopped timer (that's why we have `t.ticking` check in `stop()`
guoger (Tue, 04 Sep 2018 15:21:49 GMT):
you can think both `stop()` and `start()` as idempotent
guoger (Tue, 04 Sep 2018 15:22:12 GMT):
start a started timer - no effect
stop a stopped timer - no effect
kostas (Tue, 04 Sep 2018 15:22:26 GMT):
Roger. I think I get you now.
kostas (Tue, 04 Sep 2018 15:23:06 GMT):
Since I have your attention:
kostas (Tue, 04 Sep 2018 15:23:07 GMT):
https://gerrit.hyperledger.org/r/c/24919/29/orderer/consensus/etcdraft/chain.go#178
guoger (Tue, 04 Sep 2018 15:24:59 GMT):
in follow-up cr, it's changed to
```
t := c.NewTimer(0)
if !t.Stop() {
<-t.C()
}
```
guoger (Tue, 04 Sep 2018 15:25:14 GMT):
basically, we need a stopped timer at the end of this
kostas (Tue, 04 Sep 2018 15:25:55 GMT):
Roger. Line 174 in the follow-up CR reads nicely now.
guoger (Tue, 04 Sep 2018 15:26:08 GMT):
in original implementation, we most likely don't need to drain it, _unless_ `batchtimeout` is extremely small
kostas (Tue, 04 Sep 2018 15:26:08 GMT):
Have a comment on that follow-up CR and I think we're golden.
guoger (Tue, 04 Sep 2018 15:26:44 GMT):
ah, then `timer.C()` would panic
guoger (Tue, 04 Sep 2018 15:26:51 GMT):
and we select on it in main loop
guoger (Tue, 04 Sep 2018 15:27:08 GMT):
that's why we need a _stopped timer_, instead of _nil timer_
guoger (Tue, 04 Sep 2018 15:29:01 GMT):
i don't get this https://gerrit.hyperledger.org/r/c/24919/29/orderer/consensus/etcdraft/chain.go#66
done?
kostas (Tue, 04 Sep 2018 15:30:41 GMT):
Oh, a snafu when using Gerrit - meaning "Done", you've addressed the comment from patchset 28.
kostas (Tue, 04 Sep 2018 15:30:57 GMT):
> ah, then `timer.C()` would panic
Got it. Thanks!
guoger (Tue, 04 Sep 2018 15:32:07 GMT):
oh I see
yacovm (Tue, 04 Sep 2018 15:48:03 GMT):
why do we need raft algorithm specific stuff in `orderer.yaml` ?
yacovm (Tue, 04 Sep 2018 15:48:03 GMT):
let's talk about configuraiton
yacovm (Tue, 04 Sep 2018 15:48:03 GMT):
guys
yacovm (Tue, 04 Sep 2018 15:48:05 GMT):
we should IMO move everything that relates to raft.Node to the channel config
kostas (Tue, 04 Sep 2018 15:51:20 GMT):
I don't disagree with that. Where do you see us specifying our client TLS cert though?
kostas (Tue, 04 Sep 2018 15:51:38 GMT):
See comments here: https://gerrit.hyperledger.org/r/c/26074/3/sampleconfig/orderer.yaml
yacovm (Tue, 04 Sep 2018 15:52:19 GMT):
yeah I commented just now
yacovm (Tue, 04 Sep 2018 15:52:36 GMT):
IMO we need to expand the global config to have a client TLS cert/key
yacovm (Tue, 04 Sep 2018 15:52:43 GMT):
I think we can do that later though
yacovm (Tue, 04 Sep 2018 15:53:06 GMT):
I _ think _ cryptogen produces TLS certs that have both the server and client x509 extensions set
yacovm (Tue, 04 Sep 2018 15:53:11 GMT):
so we can use the same one for now ;)
yacovm (Tue, 04 Sep 2018 15:53:21 GMT):
(use the server cert as a client cert)
kostas (Tue, 04 Sep 2018 15:55:22 GMT):
Commented there right now, that works for me just fine and I think it's a good idea.
kostas (Tue, 04 Sep 2018 15:55:42 GMT):
We do need to figure out how to create Raft-specific defaults however, see: https://gerrit.hyperledger.org/r/c/26074/3/sampleconfig/orderer.yaml#298
yacovm (Tue, 04 Sep 2018 16:06:51 GMT):
@kostas the default i think makes sense
yacovm (Tue, 04 Sep 2018 16:06:57 GMT):
is that if the client cert isn't specified
yacovm (Tue, 04 Sep 2018 16:07:06 GMT):
we can check if the server cert can be used as a client cert
kostas (Tue, 04 Sep 2018 16:07:17 GMT):
I'm not talking about certs right now, however.
yacovm (Tue, 04 Sep 2018 16:07:20 GMT):
oh
kostas (Tue, 04 Sep 2018 16:07:27 GMT):
I'm talking about all of the other settings that go into raft.Config.
yacovm (Tue, 04 Sep 2018 16:07:40 GMT):
I have no clue sadly about defaults but you're right
kostas (Tue, 04 Sep 2018 16:07:54 GMT):
We need a template with sensible defaults stored somewhere in the global configuration. (Global as in shared across the network.)
kostas (Tue, 04 Sep 2018 16:09:06 GMT):
I think we should proceed with expanding the namespace under "EtcdRaft" in the config section. Hopefully this won't bea slippery slope. @jyellick if you have thoughts on this - see: https://gerrit.hyperledger.org/r/c/26074/3/sampleconfig/orderer.yaml#298
yacovm (Tue, 04 Sep 2018 16:09:10 GMT):
why not simply defaults puts into configtxlator?
yacovm (Tue, 04 Sep 2018 16:09:21 GMT):
or in configtx.yaml ?
kostas (Tue, 04 Sep 2018 16:09:51 GMT):
Right, but _where_ in configtx.yaml? Where do these values reside in the channel config namespace?
yacovm (Tue, 04 Sep 2018 16:10:01 GMT):
similar to Kafka no?
kostas (Tue, 04 Sep 2018 16:10:17 GMT):
Kafka is a bad citizen and pollutes the namespace.
yacovm (Tue, 04 Sep 2018 16:10:25 GMT):
huh?
yacovm (Tue, 04 Sep 2018 16:10:41 GMT):
is that how you talk about your baby?
kostas (Tue, 04 Sep 2018 16:11:04 GMT):
I subscribe to the tiger mom school of thought.
kostas (Tue, 04 Sep 2018 16:11:11 GMT):
Seriously, though: it litters the namespace.
yacovm (Tue, 04 Sep 2018 16:11:15 GMT):
I don't understand why it pollutes, it's under the orderer section
kostas (Tue, 04 Sep 2018 16:12:07 GMT):
Let me rephrase:
kostas (Tue, 04 Sep 2018 16:15:34 GMT):
As I'm about to type, I realize that my point is weaker than I thought.
kostas (Tue, 04 Sep 2018 16:16:15 GMT):
Perhaps this is a better way to put it:
kostas (Tue, 04 Sep 2018 16:16:53 GMT):
Notice how all etcd-raft settings are better encapsulated in https://gerrit.hyperledger.org/r/c/25235/4/common/tools/configtxgen/localconfig/config.go compared to their Kafka counterparts.
kostas (Tue, 04 Sep 2018 16:19:53 GMT):
If Kafka and Raft are plugins, they should spill over as less as possible to the rest of Fabric.
kostas (Tue, 04 Sep 2018 16:22:13 GMT):
(If you say, but they spill over tremendously already, you'd be right. But for the Raft case that is because of how a Fabric user absolutely needs protolator if they're to use our system. This why we did this for instance: https://gerrit.hyperledger.org/r/c/25235/4/protos/orderer/etcdraft/configuration.go#22)
kostas (Tue, 04 Sep 2018 16:23:51 GMT):
To the extend we can minimize spill over, we should. In my mind, this comes down to extending the `Metadata` message here: https://gerrit.hyperledger.org/r/c/25235/4/protos/orderer/etcdraft/configuration.proto
adarshsaraf123 (Tue, 04 Sep 2018 16:30:02 GMT):
I think I am getting what you are saying @kostas
A better perspective on this can be seen from the following Orderer interface in the channelconfig api (`common/channelconfig/api.go#81`):
```
// Orderer stores the common shared orderer config
type Orderer interface {
// ConsensusType returns the configured consensus type
ConsensusType() string
// ConsensusMetadata returns the metadata associated with the consensus type.
ConsensusMetadata() []byte
// BatchSize returns the maximum number of messages to include in a block
BatchSize() *ab.BatchSize
// BatchTimeout returns the amount of time to wait before creating a batch
BatchTimeout() time.Duration
// MaxChannelsCount returns the maximum count of channels to allow for an ordering network
MaxChannelsCount() uint64
// KafkaBrokers returns the addresses (IP:port notation) of a set of "bootstrap"
// Kafka brokers, i.e. this is not necessarily the entire set of Kafka brokers
// used for ordering
KafkaBrokers() []string
// Organizations returns the organizations for the ordering service
Organizations() map[string]Org
// Capabilities defines the capabilities for the orderer portion of a channel
Capabilities() OrdererCapabilities
}
```
Here I guess it would definitely look a lot neater if the `KafkaBrokers()` method was instead provided from `ConsensusMetadata()`.
adarshsaraf123 (Tue, 04 Sep 2018 16:30:02 GMT):
I think I am getting what you are saying @kostas
A better perspective on this can perhaps be seen from the following Orderer interface in the channelconfig api (`common/channelconfig/api.go#81`):
```
// Orderer stores the common shared orderer config
type Orderer interface {
// ConsensusType returns the configured consensus type
ConsensusType() string
// ConsensusMetadata returns the metadata associated with the consensus type.
ConsensusMetadata() []byte
// BatchSize returns the maximum number of messages to include in a block
BatchSize() *ab.BatchSize
// BatchTimeout returns the amount of time to wait before creating a batch
BatchTimeout() time.Duration
// MaxChannelsCount returns the maximum count of channels to allow for an ordering network
MaxChannelsCount() uint64
// KafkaBrokers returns the addresses (IP:port notation) of a set of "bootstrap"
// Kafka brokers, i.e. this is not necessarily the entire set of Kafka brokers
// used for ordering
KafkaBrokers() []string
// Organizations returns the organizations for the ordering service
Organizations() map[string]Org
// Capabilities defines the capabilities for the orderer portion of a channel
Capabilities() OrdererCapabilities
}
```
Here I guess it would definitely look a lot neater if the `KafkaBrokers()` method was instead provided from `ConsensusMetadata()`.
kostas (Tue, 04 Sep 2018 16:30:23 GMT):
Ah, right - that's a great example.
kostas (Tue, 04 Sep 2018 16:30:40 GMT):
There's no reason why `KafkaBrokers` should be part of that interface.
adarshsaraf123 (Tue, 04 Sep 2018 16:32:16 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=KW7QgGjS28xRmfNta) @kostas And the same also suggests that the Raft config should be part of the `Metadata` message as you suggested so that they can be provided by the `ConsensusMetadata` method.
adarshsaraf123 (Tue, 04 Sep 2018 16:32:16 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=KW7QgGjS28xRmfNta) @kostas And the same, IMO, also suggests that the Raft config should be part of the `Metadata` message as you suggested so that they can be provided by the `ConsensusMetadata` method.
kostas (Tue, 04 Sep 2018 16:33:15 GMT):
Spot on, yes. Would you like to tackle this?
adarshsaraf123 (Tue, 04 Sep 2018 16:33:35 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=3mTnDf8bGzz6pnaM6) @kostas Sounds fun ;)
kostas (Tue, 04 Sep 2018 16:33:38 GMT):
This should be a story under FAB-11474.
kostas (Tue, 04 Sep 2018 16:33:38 GMT):
This should be a ~story~ sub-task under FAB-11474.
kostas (Tue, 04 Sep 2018 16:33:42 GMT):
Excellent, thank you.
adarshsaraf123 (Tue, 04 Sep 2018 16:53:59 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=KW7QgGjS28xRmfNta) @kostas I will be leaving out the TLS config part out of this task since that will better be handled in the way @yacovm suggested (as part of the global config).
kostas (Tue, 04 Sep 2018 19:50:27 GMT):
> IMO we need to expand the global config to have a client TLS cert/key
> I think we can do that later though
> I _ think _ cryptogen produces TLS certs that have both the server and client x509 extensions set
> so we can use the same one for now 😉
> (use the server cert as a client cert)
kostas (Tue, 04 Sep 2018 19:51:20 GMT):
This is a good idea.
kostas (Tue, 04 Sep 2018 19:52:27 GMT):
I've created a story with backlogged items, this ^^ is one of them: https://jira.hyperledger.org/browse/FAB-11864
yacovm (Tue, 04 Sep 2018 19:56:39 GMT):
I'll take it,
yacovm (Tue, 04 Sep 2018 19:56:55 GMT):
> See: https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=ivh9uMbvMpb7RqfLE
@kostas :joy:
yacovm (Tue, 04 Sep 2018 19:56:55 GMT):
> See: https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=ivh9uMbvMpb7RqfLE
:joy:
yacovm (Tue, 04 Sep 2018 19:56:55 GMT):
> See: `https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=ivh9uMbvMpb7RqfLE`
:joy:
kostas (Tue, 04 Sep 2018 20:16:10 GMT):
https://jira.hyperledger.org/browse/FAB-11861?focusedCommentId=49853&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-49853
kostas (Tue, 04 Sep 2018 20:16:24 GMT):
(For anyone who may have thoughts on this.)
yacovm (Tue, 04 Sep 2018 20:18:42 GMT):
@kostas these are very good questions
yacovm (Tue, 04 Sep 2018 20:18:53 GMT):
but i don't think it's that risky...
yacovm (Tue, 04 Sep 2018 20:19:05 GMT):
if the cluster all commits the config update
yacovm (Tue, 04 Sep 2018 20:19:14 GMT):
then more than a half of it committed it
yacovm (Tue, 04 Sep 2018 20:19:27 GMT):
and it can then restart itself safely and join
yacovm (Tue, 04 Sep 2018 20:19:38 GMT):
you'll have a temporary hiccup of the orderer
yacovm (Tue, 04 Sep 2018 20:19:44 GMT):
but it should recover IMO
kostas (Tue, 04 Sep 2018 20:20:16 GMT):
Hm, good point.
kostas (Tue, 04 Sep 2018 20:20:30 GMT):
I think you're right.
yacovm (Tue, 04 Sep 2018 20:21:32 GMT):
I think this should totally be customizable though
yacovm (Tue, 04 Sep 2018 20:22:04 GMT):
I can envision someone somewhere a year from now spending an entire weekend on a production problem that can be solved by updating these stuff on some particular channel
guoger (Wed, 05 Sep 2018 07:26:05 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=k3qrH2pRT9cFHNKGr) @kostas https://jira.hyperledger.org/browse/FAB-11861?focusedCommentId=49872&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-49872
adarshsaraf123 (Wed, 05 Sep 2018 13:58:39 GMT):
@C0rWin The CR for enabling type-config updates:
https://gerrit.hyperledger.org/r/#/c/26104/
C0rWin (Wed, 05 Sep 2018 14:11:57 GMT):
@adarshsaraf123 appreciate it
guoger (Wed, 05 Sep 2018 14:32:02 GMT):
temporarily hardcoded raft configs in https://gerrit.hyperledger.org/r/c/25964, let me know if it unblocks you @yacovm
guoger (Wed, 05 Sep 2018 14:32:31 GMT):
@C0rWin addressed your comment in https://gerrit.hyperledger.org/r/c/24919 hope it makes sense
yacovm (Wed, 05 Sep 2018 14:33:33 GMT):
Thanks @guoger
yacovm (Wed, 05 Sep 2018 14:34:06 GMT):
election tick
yacovm (Wed, 05 Sep 2018 14:34:11 GMT):
what units is that?
guoger (Wed, 05 Sep 2018 14:34:39 GMT):
int
yacovm (Wed, 05 Sep 2018 14:34:44 GMT):
hmmmm ok
yacovm (Wed, 05 Sep 2018 14:35:07 GMT):
was worried for a second that it's time.Duration ;)
yacovm (Wed, 05 Sep 2018 14:35:14 GMT):
and that you made it 10 nano seconds
guoger (Wed, 05 Sep 2018 14:36:15 GMT):
super responsive cluster :metal_tone2:
C0rWin (Wed, 05 Sep 2018 14:38:10 GMT):
> @C0rWin addressed your comment in https://gerrit.hyperledger.org/r/c/24919 hope it makes sense
I think that it's not very clean to declare APIs where used only in UT. I think you can actually see whenever current node is a leader by asserting whenever it writes a block instead of adding a channel
C0rWin (Wed, 05 Sep 2018 14:38:19 GMT):
you can do it with mocks or something
guoger (Wed, 05 Sep 2018 14:45:47 GMT):
the minor problem in _raftexample_ is:
it uses `lastIndex` [here](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/raft.go#L180) to determine if replay has finished. `lastIndex` is inferred from *the Index of last entry in wal* [here](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/raft.go#L245). Although, entries in wal are unstable (may be overwritten by newer leader). Therefore, recovered former leader would potentially be blocked at [here](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/kvstore.go#L43).
To reproduce:
- start raftexample 3-nodes cluster with long TickInterval (change code and recompile), so you have enough time to send requests while it's in transition)
- send some requests to leader
- kill two _followers_
- send more requests to _leader_ (so it has more uncommitted entries appended in wal than other nodes)
- kill _leader_
- restart two _followers_, wait for a leader to be elected
- restart _leader_, it would be stuck at the line aforementioned
guoger (Wed, 05 Sep 2018 14:45:47 GMT):
the minor problem in _raftexample_ I mentioned during the meeting is:
it uses `lastIndex` [here](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/raft.go#L180) to determine if replay has finished. `lastIndex` is inferred from *the Index of last entry in wal* [here](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/raft.go#L245). Although, entries in wal are unstable (may be overwritten by newer leader). Therefore, recovered former leader would potentially be blocked at [here](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/kvstore.go#L43).
To reproduce:
- start raftexample 3-nodes cluster with long TickInterval (change code and recompile), so you have enough time to send requests while it's in transition)
- send some requests to leader
- kill two _followers_
- send more requests to _leader_ (so it has more uncommitted entries appended in wal than other nodes)
- kill _leader_
- restart two _followers_, wait for a leader to be elected
- restart _leader_, it would be stuck at the line aforementioned
guoger (Wed, 05 Sep 2018 14:48:09 GMT):
cluster would still function correctly after sending more transactions to new leader, and wait for them to overwrite stale entries on former leader, till `Index` exceeds the _last index_ previously persisted in former leader.
guoger (Wed, 05 Sep 2018 14:50:15 GMT):
*tl;dr*, the code in _raftexample_ is correct but written in an odd way. we should follow etcd/swarmkit w.r.t `wal`
guoger (Wed, 05 Sep 2018 14:52:31 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=Fujo7XepHRGFffYFx) @C0rWin tried it, and it's neither clean nor testing what we intended. pls feel free to try cleaning it up, but my suggestion is let's proceed with this
C0rWin (Wed, 05 Sep 2018 14:54:18 GMT):
> @C0rWin tried it, and it's neither clean nor testing what we intended. pls feel free to try cleaning it up, but my suggestion is let's proceed with this
I cannot say I agree with this, but as far as @sykesm and @kostas considered this as acceptable, I will remove my -1 for now, to not block this from going in and we can definitely address it later
guoger (Wed, 05 Sep 2018 15:00:58 GMT):
yup, that's what I was trying to say, let's not clog pipeline with this and we could always revisit this
C0rWin (Wed, 05 Sep 2018 15:01:42 GMT):
sure
guoger (Wed, 05 Sep 2018 15:05:05 GMT):
(I did try using various way of testing it without adding channel, but tests end up being flaky or poorly-readable, but I *do* appreciate any clean-up that I didn't think of. For multi-cluster tests in particular, we do need a means to detect leader change deterministically)
kostas (Thu, 06 Sep 2018 00:46:53 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=qRESNebBjdjpCGEqx
kostas (Thu, 06 Sep 2018 00:47:01 GMT):
Excellent write-up, thanks Jay.
kostas (Thu, 06 Sep 2018 00:47:51 GMT):
I had missed that. Good catch!
kostas (Thu, 06 Sep 2018 00:54:34 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=hpgCP7A7jPix49yGT
kostas (Thu, 06 Sep 2018 00:56:13 GMT):
FWIW, I totally missed that in my review. Artem's right to call that out. But for now, yes, let's clear up the pipeline a bit and if anybody thinks of a better way to handle this, let's go for it.
kostas (Thu, 06 Sep 2018 13:09:53 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=MAmhRqMmi7Z4vba87
kostas (Thu, 06 Sep 2018 13:10:10 GMT):
@guoger: Not sure we'd want to spill over to the localconfig package for this?
kostas (Thu, 06 Sep 2018 13:10:23 GMT):
Why not hard-code them inside "etcdraft"?
yacovm (Thu, 06 Sep 2018 13:22:15 GMT):
@guoger how close are you to make the FSM support multi-nodes? :thinking:
yacovm (Thu, 06 Sep 2018 14:38:53 GMT):
@kostas
yacovm (Thu, 06 Sep 2018 14:39:05 GMT):
why are we reading the file system inside the `New` of the consenter?
yacovm (Thu, 06 Sep 2018 14:39:11 GMT):
we should inject dependenices
yacovm (Thu, 06 Sep 2018 14:39:11 GMT):
we should inject dependencies
yacovm (Thu, 06 Sep 2018 14:39:18 GMT):
and not read them from the file system :(
yacovm (Thu, 06 Sep 2018 14:39:47 GMT):
we have a reference to the certificate in the `main.go` and we can just pass it as a reference to the consenter...
kostas (Thu, 06 Sep 2018 14:39:47 GMT):
Link?
yacovm (Thu, 06 Sep 2018 14:40:17 GMT):
https://gerrit.hyperledger.org/r/#/c/25964/9/orderer/consensus/etcdraft/consenter.go@32
kostas (Thu, 06 Sep 2018 14:41:44 GMT):
I think that's a good call. Add it to the CR as comment?
kostas (Thu, 06 Sep 2018 14:41:44 GMT):
I think that's a good call. Add it to the CR as a comment?
yacovm (Thu, 06 Sep 2018 14:42:27 GMT):
no...
yacovm (Thu, 06 Sep 2018 14:42:37 GMT):
it will create a cascading conflict
yacovm (Thu, 06 Sep 2018 14:42:45 GMT):
I prefere to just address it in a CR
yacovm (Thu, 06 Sep 2018 14:42:56 GMT):
otherwise we'll never finish this work
kostas (Thu, 06 Sep 2018 14:43:18 GMT):
I'm fine either way.
yacovm (Thu, 06 Sep 2018 14:43:29 GMT):
I don't want Jay to suffer any more than he has to so I'll jut do it
yacovm (Thu, 06 Sep 2018 14:43:29 GMT):
I don't want Jay to suffer any more than he has to so I'll just do it
kostas (Thu, 06 Sep 2018 14:43:58 GMT):
Suffer?
yacovm (Thu, 06 Sep 2018 14:44:50 GMT):
constantly changing his code
guoger (Thu, 06 Sep 2018 15:19:31 GMT):
> Why not hard-code them inside "etcdraft"?
I don't understand.. expand please?
guoger (Thu, 06 Sep 2018 15:21:06 GMT):
> I don't want Jay to suffer any more than he has to so I'll just do it
haha, not suffering at all. I was suggested to read the file and I didn't think that much.
guoger (Thu, 06 Sep 2018 15:21:35 GMT):
I could update it the first thing tomorrow morning. but if you want to do it now, go for it :) @yacovm
guoger (Thu, 06 Sep 2018 15:23:01 GMT):
> how close are you to make the FSM support multi-nodes?
1) rebase the code 2) wire them to communicator
kostas (Thu, 06 Sep 2018 15:23:03 GMT):
var RaftChannelOptions = struct {
TickInterval time.Duration
ElectionTick int
HeartbeatTick int
MaxSizePerMsg uint64
MaxInflightMsgs int
}{TickInterval: ..., ElectionTick: ...}
kostas (Thu, 06 Sep 2018 15:23:03 GMT):
```var RaftChannelOptions = struct {
TickInterval time.Duration
ElectionTick int
HeartbeatTick int
MaxSizePerMsg uint64
MaxInflightMsgs int
}{TickInterval: ..., ElectionTick: ...}```
kostas (Thu, 06 Sep 2018 15:23:27 GMT):
And have that var in `chain.go`.
guoger (Thu, 06 Sep 2018 15:25:26 GMT):
and then have `type Option struct` encapsulating `Storage`, `Clock`, `RaftChannelOptions`, etc?
kostas (Thu, 06 Sep 2018 15:26:31 GMT):
That part I haven't thought about, but my main point is - if we know that these options are going to leave in Adarsh's `Metadata` message, we should contain them now temporarily in `etcdraft` instead of in any other package.
kostas (Thu, 06 Sep 2018 15:26:44 GMT):
Not a deal-breaker at the end of the day.
guoger (Thu, 06 Sep 2018 15:26:48 GMT):
> Not sure we'd want to spill over to the localconfig package for this?
if you are talking about [this](https://gerrit.hyperledger.org/r/c/25964/9/orderer/common/localconfig/config.go#165), yes, I forgot to delete it from localconfig.
kostas (Thu, 06 Sep 2018 15:26:49 GMT):
We know that this will change.
kostas (Thu, 06 Sep 2018 15:27:02 GMT):
Yes, this is what I was talking about.
guoger (Thu, 06 Sep 2018 15:27:53 GMT):
then absolutely corret, as Artem also pointed out in review, my bad :P
kostas (Thu, 06 Sep 2018 15:28:43 GMT):
No worries
yacovm (Thu, 06 Sep 2018 15:57:28 GMT):
Noo jay dont wire into the communicator yet 😉
yacovm (Thu, 06 Sep 2018 15:57:52 GMT):
I just dont like doing file system io inside constructora
huikang (Thu, 06 Sep 2018 16:06:48 GMT):
Hi, since Jay's Raft chain change has been merged, is there any plan to support reconfiguring raft on top of that? I am interesting in doing that. Thanks.
kostas (Thu, 06 Sep 2018 19:30:59 GMT):
@huikang: Hello! Haven't forgotten about your offer to help - and thanks for that! Unfortunately I can't think of any additional task that can be processed in parallel _right now_, and I don't want to have you running in circles. As thing stand right now: Jay's working on adding crash-fault tolerance to the ordering node (FAB-11589), Yacov's working on wiring the components so that we can bootstrap a Raft-based ordering node (FAB-11798), and Artem (Type B) and Adarsh (Type A) are tackling reconfiguration (FAB-11590). You can help out by reviewing the code that's out there (which will help us identify deficiencies that _you_ can work on), or keep an eye out on https://jira.hyperledger.org/browse/FAB-11863 for backlogged issues. I'd also ask the team to consider spitting out additional sub-tasks, as they're working on their stories and leave them out for grabs. This should also help with creating a list of issues that can be tackled.
kostas (Thu, 06 Sep 2018 19:30:59 GMT):
@huikang: Hello! Haven't forgotten about your offer to help - and thanks for that! Unfortunately I can't think of any additional task that can be processed in parallel _right now_, and I don't want to have you running in circles. As thing stand right now: Jay's working on adding crash-fault tolerance to the ordering node (FAB-11589), Yacov's working on wiring the components so that we can bootstrap a Raft-based ordering node (FAB-11798), and Artem (Type B ) and Adarsh (Type A) are tackling reconfiguration (FAB-11590). You can help out by reviewing the code that's out there (which will help us identify deficiencies that _you_ can work on), or keep an eye out on https://jira.hyperledger.org/browse/FAB-11863 for backlogged issues. I'd also ask the team to consider spitting out additional sub-tasks, as they're working on their stories and leave them out for grabs. This should also help with creating a list of issues that can be tackled.
huikang (Thu, 06 Sep 2018 19:40:13 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=xZtpxgNK3jDCxsq7C) @kostas Hi, Kostas, sure I will be glad to review the code there and look for new tasks from there. Thanks.
guoger (Fri, 07 Sep 2018 02:40:56 GMT):
@kostas @sykesm replied in [CR](https://gerrit.hyperledger.org/r/c/25975/), IIUC, you suggestions reset the timer every time a new envelope comes in. Although we shouldn't be doing that. If a timer is started, we don't reset
yj511608130 (Fri, 07 Sep 2018 03:54:06 GMT):
Has joined the channel.
guoger (Fri, 07 Sep 2018 09:41:47 GMT):
i have an update to [this CR|https://gerrit.hyperledger.org/r/c/25964] to address comments. However, as Yacov aforementioned, it would cause cascading conflicts. How could I push it somewhere so it can be reviewed independently, and once it's ready, following CRs in stack only need to be rebased once.
guoger (Fri, 07 Sep 2018 09:41:47 GMT):
i have an update to [this CR](https://gerrit.hyperledger.org/r/c/25964) to address comments. However, as Yacov aforementioned, it would cause cascading conflicts. How could I push it somewhere so it can be reviewed independently, and once it's ready, following CRs in stack only need to be rebased once.
yacovm (Fri, 07 Sep 2018 10:20:01 GMT):
it's impossible to do what you ask...
yacovm (Fri, 07 Sep 2018 10:20:07 GMT):
just address in that change set
yacovm (Fri, 07 Sep 2018 10:20:19 GMT):
and I'll rebase on top once it was reviewed
raviyelleni (Sun, 09 Sep 2018 04:44:36 GMT):
Has joined the channel.
yacovm (Sun, 09 Sep 2018 22:49:39 GMT):
Everyone - we have a working Raft OSN - https://gerrit.hyperledger.org/r/#/c/26038/ ( @kostas @C0rWin @guoger )
```
2018-09-10 01:46:27.816 IDT [orderer/common/server] initializeServerConfig -> INFO 003 Starting orderer with mutual TLS enabled
2018-09-10 01:46:27.826 IDT [fsblkstorage] newBlockfileMgr -> INFO 004 Getting block information from block storage
2018-09-10 01:46:27.848 IDT [orderer/commmon/multichannel] Initialize -> INFO 005 Starting system channel 'system' with genesis block hash e37db8b8a56929b375adc05f9961fe681d39f45262d61babbe5e1038dd2c2681 and orderer type etcdraft
2018-09-10 01:46:27.849 IDT [orderer/consensus/etcdraft] becomeFollower -> INFO 006 1 became follower at term 0 {"channel": "system"}
2018-09-10 01:46:27.849 IDT [orderer/consensus/etcdraft] newRaft -> INFO 007 newRaft 1 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0] {"channel": "system"}
2018-09-10 01:46:27.849 IDT [orderer/consensus/etcdraft] becomeFollower -> INFO 008 1 became follower at term 1 {"channel": "system"}
2018-09-10 01:46:27.849 IDT [orderer/common/cluster] Configure -> INFO 009 Entering, channel: system, nodes: []
2018-09-10 01:46:27.849 IDT [orderer/common/cluster] Configure -> INFO 00a Exiting
2018-09-10 01:46:27.849 IDT [orderer/common/server] Start -> INFO 00b Starting orderer:
Version: 1.3.0
Commit SHA: 04e3af2
Go version: go1.10.3
OS/Arch: linux/amd64
Experimental features: true
2018-09-10 01:46:27.849 IDT [orderer/common/server] Start -> INFO 00c Beginning to serve requests
2018-09-10 01:46:28.950 IDT [orderer/consensus/etcdraft] Step -> INFO 00d 1 is starting a new election at term 1 {"channel": "system"}
2018-09-10 01:46:28.950 IDT [orderer/consensus/etcdraft] becomeCandidate -> INFO 00e 1 became candidate at term 2 {"channel": "system"}
2018-09-10 01:46:28.950 IDT [orderer/consensus/etcdraft] poll -> INFO 00f 1 received MsgVoteResp from 1 at term 2 {"channel": "system"}
2018-09-10 01:46:28.950 IDT [orderer/consensus/etcdraft] becomeLeader -> INFO 010 1 became leader at term 2 {"channel": "system"}
2018-09-10 01:46:28.950 IDT [orderer/consensus/etcdraft] run -> INFO 011 raft.node: 1 elected leader 1 at term 2 {"channel": "system"}
2018-09-10 01:46:28.951 IDT [orderer/consensus/etcdraft] serveRaft -> INFO 012 Raft leader changed on node 1: 0 -> 1 {"channel": "system"}
```
@guoger can you please rebase your change sets above the change set? It touches quite a bit of places and it is crucial to keep it functioning so I can proceed with integration tests ...
yacovm (Sun, 09 Sep 2018 22:49:39 GMT):
Everyone - we have a working Raft OSN - https://gerrit.hyperledger.org/r/#/c/26038/ ( @kostas @C0rWin @guoger )
```
2018-09-10 01:46:27.816 IDT [orderer/common/server] initializeServerConfig -> INFO 003 Starting orderer with mutual TLS enabled
2018-09-10 01:46:27.826 IDT [fsblkstorage] newBlockfileMgr -> INFO 004 Getting block information from block storage
2018-09-10 01:46:27.848 IDT [orderer/commmon/multichannel] Initialize -> INFO 005 Starting system channel 'system' with genesis block hash e37db8b8a56929b375adc05f9961fe681d39f45262d61babbe5e1038dd2c2681 and orderer type etcdraft
2018-09-10 01:46:27.849 IDT [orderer/consensus/etcdraft] becomeFollower -> INFO 006 1 became follower at term 0 {"channel": "system"}
2018-09-10 01:46:27.849 IDT [orderer/consensus/etcdraft] newRaft -> INFO 007 newRaft 1 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0] {"channel": "system"}
2018-09-10 01:46:27.849 IDT [orderer/consensus/etcdraft] becomeFollower -> INFO 008 1 became follower at term 1 {"channel": "system"}
2018-09-10 01:46:27.849 IDT [orderer/common/cluster] Configure -> INFO 009 Entering, channel: system, nodes: []
2018-09-10 01:46:27.849 IDT [orderer/common/cluster] Configure -> INFO 00a Exiting
2018-09-10 01:46:27.849 IDT [orderer/common/server] Start -> INFO 00b Starting orderer:
Version: 1.3.0
Commit SHA: 04e3af2
Go version: go1.10.3
OS/Arch: linux/amd64
Experimental features: true
2018-09-10 01:46:27.849 IDT [orderer/common/server] Start -> INFO 00c Beginning to serve requests
2018-09-10 01:46:28.950 IDT [orderer/consensus/etcdraft] Step -> INFO 00d 1 is starting a new election at term 1 {"channel": "system"}
2018-09-10 01:46:28.950 IDT [orderer/consensus/etcdraft] becomeCandidate -> INFO 00e 1 became candidate at term 2 {"channel": "system"}
2018-09-10 01:46:28.950 IDT [orderer/consensus/etcdraft] poll -> INFO 00f 1 received MsgVoteResp from 1 at term 2 {"channel": "system"}
2018-09-10 01:46:28.950 IDT [orderer/consensus/etcdraft] becomeLeader -> INFO 010 1 became leader at term 2 {"channel": "system"}
2018-09-10 01:46:28.950 IDT [orderer/consensus/etcdraft] run -> INFO 011 raft.node: 1 elected leader 1 at term 2 {"channel": "system"}
2018-09-10 01:46:28.951 IDT [orderer/consensus/etcdraft] serveRaft -> INFO 012 Raft leader changed on node 1: 0 -> 1 {"channel": "system"}
```
@guoger can you please rebase your change sets above the change set? It touches quite a bit of places and it is crucial to keep it functioning so I can proceed with integration tests ...
yacovm (Sun, 09 Sep 2018 22:49:39 GMT):
Everyone - we have a working Raft OSN - https://gerrit.hyperledger.org/r/#/c/26038/ ( @kostas @C0rWin @guoger )
```
2018-09-10 01:46:27.816 IDT [orderer/common/server] initializeServerConfig -> INFO 003 Starting orderer with mutual TLS enabled
2018-09-10 01:46:27.826 IDT [fsblkstorage] newBlockfileMgr -> INFO 004 Getting block information from block storage
2018-09-10 01:46:27.848 IDT [orderer/commmon/multichannel] Initialize -> INFO 005 Starting system channel 'system' with genesis block hash e37db8b8a56929b375adc05f9961fe681d39f45262d61babbe5e1038dd2c2681 and orderer type etcdraft
2018-09-10 01:46:27.849 IDT [orderer/consensus/etcdraft] becomeFollower -> INFO 006 1 became follower at term 0 {"channel": "system"}
2018-09-10 01:46:27.849 IDT [orderer/consensus/etcdraft] newRaft -> INFO 007 newRaft 1 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0] {"channel": "system"}
2018-09-10 01:46:27.849 IDT [orderer/consensus/etcdraft] becomeFollower -> INFO 008 1 became follower at term 1 {"channel": "system"}
2018-09-10 01:46:27.849 IDT [orderer/common/cluster] Configure -> INFO 009 Entering, channel: system, nodes: []
2018-09-10 01:46:27.849 IDT [orderer/common/cluster] Configure -> INFO 00a Exiting
2018-09-10 01:46:27.849 IDT [orderer/common/server] Start -> INFO 00b Starting orderer:
Version: 1.3.0
Commit SHA: 04e3af2
Go version: go1.10.3
OS/Arch: linux/amd64
Experimental features: true
2018-09-10 01:46:27.849 IDT [orderer/common/server] Start -> INFO 00c Beginning to serve requests
2018-09-10 01:46:28.950 IDT [orderer/consensus/etcdraft] Step -> INFO 00d 1 is starting a new election at term 1 {"channel": "system"}
2018-09-10 01:46:28.950 IDT [orderer/consensus/etcdraft] becomeCandidate -> INFO 00e 1 became candidate at term 2 {"channel": "system"}
2018-09-10 01:46:28.950 IDT [orderer/consensus/etcdraft] poll -> INFO 00f 1 received MsgVoteResp from 1 at term 2 {"channel": "system"}
2018-09-10 01:46:28.950 IDT [orderer/consensus/etcdraft] becomeLeader -> INFO 010 1 became leader at term 2 {"channel": "system"}
2018-09-10 01:46:28.950 IDT [orderer/consensus/etcdraft] run -> INFO 011 raft.node: 1 elected leader 1 at term 2 {"channel": "system"}
2018-09-10 01:46:28.951 IDT [orderer/consensus/etcdraft] serveRaft -> INFO 012 Raft leader changed on node 1: 0 -> 1 {"channel": "system"}
```
@guoger can you please rebase your change sets (that are in progress, and are not ancestors of my change set) above the change set? It touches quite a bit of places and it is crucial to keep it functioning so I can proceed with integration tests ...
C0rWin (Sun, 09 Sep 2018 22:51:09 GMT):
:clap:
guoger (Mon, 10 Sep 2018 01:41:52 GMT):
:metal_tone2:
guoger (Mon, 10 Sep 2018 01:42:27 GMT):
those two in-review CRs are for crash and recovery, are you testing those already?
guoger (Mon, 10 Sep 2018 01:51:33 GMT):
(I certainly could rebase on top yours, just curious about the urgency)
guoger (Mon, 10 Sep 2018 01:51:33 GMT):
(I certainly should rebase on top yours, just curious about the urgency)
guoger (Mon, 10 Sep 2018 05:29:27 GMT):
I'm not sure about the term in https://gerrit.hyperledger.org/r/c/26178. In etcd/raft, FSM has two [meanings](https://github.com/etcd-io/etcd/tree/master/raft):
- the state machine that consented messages are applied on, i.e. _Apply Snapshot (if any) and CommittedEntries to the state machine_
- etcd/raft internal state machine, i.e. _the library models Raft as a state machine_
to distinguish, I would suggest `RaftConfig` vs `FabricConfig`/`ConsenterConfig`. wdyt, @adarshsaraf123
adarshsaraf123 (Mon, 10 Sep 2018 05:52:33 GMT):
@guoger Sure we can try a different term. I couldn't reply to your comment on the CR but I also like just Config as you had suggested.
yacovm (Mon, 10 Sep 2018 09:30:49 GMT):
@guoger - no... i am not testing anything. I just want to make sure we don't have a jungle of change sets and we'll end up in conflict resolution hell.
It's not urgent... rebase when you think your stuff is ready
yacovm (Mon, 10 Sep 2018 09:33:00 GMT):
@guoger m @adarshsaraf123 maybe instead of FSM we can call it RSM ? as in Replicated state machine (not research staff member)
yacovm (Mon, 10 Sep 2018 09:33:21 GMT):
this is essentially, the configuration of the replicated state machine, no?
adarshsaraf123 (Mon, 10 Sep 2018 09:40:05 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=GEXtrHQdPn2GddekC) @yacovm Posted similar comment on the CR. I am only wondering if when we read the from within the code will it be instructive enough to have just `etcdraft.RSM` and that's why had started out with `etcdraft.FSMConfig`. WDYT?
yacovm (Mon, 10 Sep 2018 09:40:55 GMT):
well we have `Consenters` don't we?
yacovm (Mon, 10 Sep 2018 09:41:04 GMT):
doesn't the same argument apply here?
guoger (Mon, 10 Sep 2018 09:46:33 GMT):
I'm still in favor of RaftConfig:
- OSNs are RSM, etcd/raft is RSM too
- RaftConfig is smaller than RSMConfig in scope
adarshsaraf123 (Mon, 10 Sep 2018 09:50:59 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=fTCzdPR6jhspCGnoC) @yacovm WRT `etcdraft.Consenters` personally I think more of the struct members for each `Consenter` as identifying the consenter and not just as configuration value for the consenter. In the case of the `etcdraft.RSM` the struct members do not seem to identify an RSM but only specify some configuration parameters for the RSM.
yacovm (Mon, 10 Sep 2018 09:51:13 GMT):
sure.... if you insist. it doesn't matter to me. I'm fine calling it `Foo` as well.
adarshsaraf123 (Mon, 10 Sep 2018 09:52:33 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=wmQdxYBdncNz46S6T) @guoger I think the `Raft` in `RaftConfig` is implied from the import since it will be `etcdraft.RaftConfig`. This is why I had liked just `etcdraft.Config` suggested by you in the CR.
adarshsaraf123 (Mon, 10 Sep 2018 10:41:59 GMT):
@yacovm @guoger How about we call it `etcdraft.Params`? Since @yacovm 's point about `Config` not looking good in `configtx.yaml` is also very true.
yacovm (Mon, 10 Sep 2018 10:44:39 GMT):
I personally think that `Params` is too general and I think `RSM` best describes what the parameters are, but - as I said - it doesn't matter to me that much.
dairehoman (Mon, 10 Sep 2018 19:09:38 GMT):
Has joined the channel.
adarshsaraf123 (Mon, 10 Sep 2018 19:45:23 GMT):
On going through the naming patterns in the `sampleconfig/*.yaml` files, I realize that @yacovm 's proposal of the name being `RSM` makes most sense.
guoger (Tue, 11 Sep 2018 00:21:09 GMT):
Why is it?
kostas (Tue, 11 Sep 2018 03:18:12 GMT):
I'm slightly partial to `RaftConfig` but I hear the argument about stuttering.
huikang (Tue, 11 Sep 2018 03:26:41 GMT):
how about* etcdraft.ClusterConfig* if the parameters are applied to the raft cluster?
huikang (Tue, 11 Sep 2018 03:26:41 GMT):
how about *etcdraft.ClusterConfig* if the parameters are applied to the raft cluster?
adarshsaraf123 (Tue, 11 Sep 2018 04:44:10 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=yP80dW7acE0pPiLdo0) @guoger If you look at all the key names there, they all specify various configuration parameters and none of them have the name `Config` appended to them
adarshsaraf123 (Tue, 11 Sep 2018 04:45:24 GMT):
So I guess it will be good if we can avoid having `Config` in the name and since @guoger mentioned that `FSM` seems to generic I had thought `RSM` was better.
guoger (Tue, 11 Sep 2018 04:52:21 GMT):
yes, and in that case why not RaftConfig, just to be specific. I don't think we'll ever have consensus on this name, so I'm OK with whatever you feel most comfortable with :)
adarshsaraf123 (Tue, 11 Sep 2018 04:55:36 GMT):
Yaa consensus _is_ a *hard* problem.
adarshsaraf123 (Tue, 11 Sep 2018 04:55:36 GMT):
Yaa consensus _is a hard problem_.
adarshsaraf123 (Tue, 11 Sep 2018 04:55:36 GMT):
Yaa consensus _is a hard problem_.
yacovm (Tue, 11 Sep 2018 07:14:22 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=fHXmSYStjMyZ9zyee
yacovm (Tue, 11 Sep 2018 07:14:45 GMT):
we already have a cluster config in the code so we can't reuse it
yacovm (Tue, 11 Sep 2018 07:49:34 GMT):
you've been typing for an hour, Jay
yacovm (Tue, 11 Sep 2018 07:49:41 GMT):
I addressed your comments
guoger (Tue, 11 Sep 2018 07:50:52 GMT):
@yacovm pls help me understand communication, thx!
- `Chain` should be using `RemoteContext` to send msg to other nodes, right?
- what if a connection cannot be established, would `Communicator.Remote` fail? or it still returns an obj, but subsequent calls fail?
yacovm (Tue, 11 Sep 2018 07:51:08 GMT):
no
yacovm (Tue, 11 Sep 2018 07:51:23 GMT):
you just use the `RPC` struct to send to remote nodes
yacovm (Tue, 11 Sep 2018 07:51:47 GMT):
`RPC` is just a friendly wrapper around the comm
guoger (Tue, 11 Sep 2018 07:53:59 GMT):
and how do i get rpc?
yacovm (Tue, 11 Sep 2018 07:54:10 GMT):
to initialize an `RPC` you just create the struct instance and pass the reference of `Comm` into it.
To wire the reference of the `Comm` into it, you need to simply add another exported field to the `Consenter` like I did for the `Configurator Configurator` which is the comm itself
yacovm (Tue, 11 Sep 2018 07:54:46 GMT):
I guess I can just make `Configurator` to a full blown `Communicator` which also is a `Configurator`
yacovm (Tue, 11 Sep 2018 07:54:49 GMT):
in my change set
guoger (Tue, 11 Sep 2018 07:54:56 GMT):
yeah, I was about to ask that
yacovm (Tue, 11 Sep 2018 07:54:58 GMT):
and then things will be easier for you
guoger (Tue, 11 Sep 2018 07:55:03 GMT):
either way
guoger (Tue, 11 Sep 2018 07:55:03 GMT):
~either way~ it's better that you do it
yacovm (Tue, 11 Sep 2018 07:55:03 GMT):
wdyt?
yacovm (Tue, 11 Sep 2018 07:55:13 GMT):
I'll do it now
yacovm (Tue, 11 Sep 2018 07:55:57 GMT):
but in the meantime you understand how to create an `RPC` ?
yacovm (Tue, 11 Sep 2018 07:56:02 GMT):
you just give it the name of the channel
yacovm (Tue, 11 Sep 2018 07:56:08 GMT):
and the reference of the `Comm`
guoger (Tue, 11 Sep 2018 07:57:03 GMT):
yup
guoger (Tue, 11 Sep 2018 07:57:58 GMT):
I almost wanted to call `Remote` for every message in `Chain` and manage those stubs in Chain. And I figured I better ask :joy:
yacovm (Tue, 11 Sep 2018 08:00:05 GMT):
`Remote` is smart, actually
yacovm (Tue, 11 Sep 2018 08:00:22 GMT):
it returns to you an existing connection if it exists
guoger (Tue, 11 Sep 2018 08:05:13 GMT):
wait, I thought RPC does that for me, no?
guoger (Tue, 11 Sep 2018 08:05:13 GMT):
~wait, I thought RPC does that for me, no?~
guoger (Tue, 11 Sep 2018 08:11:42 GMT):
nvm, misread msg
yacovm (Tue, 11 Sep 2018 08:31:18 GMT):
@guoger done
Rosan (Thu, 13 Sep 2018 06:42:24 GMT):
Has joined the channel.
guoger (Thu, 13 Sep 2018 09:01:05 GMT):
why are both of [this](https://github.com/hyperledger/fabric/blob/master/sampleconfig/configtx.yaml#L523-L524) using client cert?
guoger (Thu, 13 Sep 2018 09:14:49 GMT):
@yacovm may I know which integration scenario are you currently working on? trying to avoid overlap here. thx
yacovm (Thu, 13 Sep 2018 09:25:10 GMT):
I'm working on a single raft node for now
yacovm (Thu, 13 Sep 2018 09:25:30 GMT):
but before I need to even make it possible to create a raft node in the `nwo` stuff
guoger (Thu, 13 Sep 2018 09:43:01 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=sSpCgTsMd5KpMTxEi) @yacovm btw, can you take a look at this when you have time?
yacovm (Thu, 13 Sep 2018 09:43:10 GMT):
i did
yacovm (Thu, 13 Sep 2018 09:43:13 GMT):
i have no clue
yacovm (Thu, 13 Sep 2018 09:43:19 GMT):
it's not important
yacovm (Thu, 13 Sep 2018 09:43:31 GMT):
it's only a file path....
guoger (Thu, 13 Sep 2018 09:45:42 GMT):
I'm asking @adarshsaraf123 to help with multi-node integration test, and I'm trying to figure out how it can be parallelized with your integration test work
guoger (Thu, 13 Sep 2018 09:45:42 GMT):
I'm asking @adarshsaraf123 to help with multi-node integration test, and I'm trying to figure out how it can be parallelized with _your_ integration test work
yacovm (Thu, 13 Sep 2018 09:46:07 GMT):
it cannot....
adarshsaraf123 (Thu, 13 Sep 2018 09:46:56 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=sSpCgTsMd5KpMTxEi) @guoger I think it is just a tiny mistake. I had stumbled upon it when doing the configuration defaults work.
adarshsaraf123 (Thu, 13 Sep 2018 09:46:56 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=sSpCgTsMd5KpMTxEi) @guoger I think it is just a tiny mistake. I had stumbled upon it when doing the configuration defaults work and was thinking of fixing it in my patch.
yacovm (Thu, 13 Sep 2018 09:50:53 GMT):
@guoger I don't understand though, you're doing now ginkgo tests for multi node raft
yacovm (Thu, 13 Sep 2018 09:51:24 GMT):
> I'm asking @adarshsaraf123 to help with multi-node integration test, and I'm trying to figure out how it can be parallelized with _your_ integration test work
just to be precise - you mean the ginkgo tests under `integration/` right?
guoger (Thu, 13 Sep 2018 09:51:43 GMT):
yes
yacovm (Thu, 13 Sep 2018 09:52:31 GMT):
well he can't really start anything until I'm done preparing the integration tests infrastructure to support raft type OSNs
yacovm (Thu, 13 Sep 2018 09:52:49 GMT):
sorry to dissapoint :)
guoger (Thu, 13 Sep 2018 09:53:55 GMT):
ha, not at all, just wanna check so we have view
guoger (Thu, 13 Sep 2018 09:54:03 GMT):
*same view
adarshsaraf123 (Thu, 13 Sep 2018 09:59:35 GMT):
@yacovm About my comment [here](https://gerrit.hyperledger.org/r/#/c/26038/34/orderer/consensus/etcdraft/consenter.go) at line 80 about having to check for all parameters of the consenter matching to detect self-id. You said that isn't necessary. Can you please comment on it further? I thought if the consenting node is not correctly configured to have the same parameters as specified in the consenters set, then the other nodes will not be able to discover this node correctly. If the host/ip is mismatched then they cannot contact this node and if the client cert is different it might fail the TLS connection.
adarshsaraf123 (Thu, 13 Sep 2018 09:59:35 GMT):
@yacovm About my comment [here](https://gerrit.hyperledger.org/r/#/c/26038/34/orderer/consensus/etcdraft/consenter.go) at line 80 about having to check for all parameters of the consenter matching to detect self-id. You said that isn't necessary. Can you please comment on it further? I thought if the consenting node is not correctly configured to have the same parameters as specified in the consenters set, then the other consenting nodes will not be able to discover this node correctly. If the host/ip is mismatched then they cannot contact this node and if the client cert is different it might fail the TLS connection.
yacovm (Thu, 13 Sep 2018 10:00:41 GMT):
but you need to make sure that _ you _ are reachable
yacovm (Thu, 13 Sep 2018 10:00:47 GMT):
not that you can connect to others
yacovm (Thu, 13 Sep 2018 10:01:09 GMT):
The orderer doesn't have an endpoint
yacovm (Thu, 13 Sep 2018 10:01:12 GMT):
it's not a peer
yacovm (Thu, 13 Sep 2018 10:01:19 GMT):
so there is no string to compare yourself to anyone
yacovm (Thu, 13 Sep 2018 10:01:28 GMT):
how do you know what is your endpoint?
yacovm (Thu, 13 Sep 2018 10:01:37 GMT):
give me an example please
adarshsaraf123 (Thu, 13 Sep 2018 10:08:05 GMT):
The etcdraft metadata contains the `Consenter` list which I believe is the only place from where the various consenting nodes come to know who the other consenting nodes are in the cluster. This `Consenter` [struct](https://github.com/hyperledger/fabric/blob/master/protos/orderer/etcdraft/configuration.proto#L21-#L26) specifies `host`, `port`, `client-tls-cert` and `server-tls-cert`. Now when I spin up a new raft node if I only compare my server cert to detect my own id while the other three of these are different, will it not happen that the other nodes will not be able to correctly locate/identify me?
adarshsaraf123 (Thu, 13 Sep 2018 10:09:15 GMT):
@yacovm ^^
yacovm (Thu, 13 Sep 2018 10:09:49 GMT):
so you proposed to check the ip/host, right?
yacovm (Thu, 13 Sep 2018 10:09:56 GMT):
> If the host/ip is mismatched then they cannot contact this node and if the client cert is different it might fail the TLS connection.
yacovm (Thu, 13 Sep 2018 10:10:08 GMT):
i'm asking - how do you know, your host / ip ?
adarshsaraf123 (Thu, 13 Sep 2018 10:10:17 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=eHyKQA7y8rB66XH9n) @yacovm Yes, and the client cert as well
adarshsaraf123 (Thu, 13 Sep 2018 10:11:03 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=e5gN8a42KzKxE7wj7) @yacovm isn't this supposed to be specified in `orderer.yaml` like the other certs.
adarshsaraf123 (Thu, 13 Sep 2018 10:11:03 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=e5gN8a42KzKxE7wj7) @yacovm isn't this supposed to be specified in `orderer.yaml` like the other certs.?
adarshsaraf123 (Thu, 13 Sep 2018 10:11:03 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=e5gN8a42KzKxE7wj7) @yacovm isn't this supposed to be specified in `orderer.yaml` like the other certs?
yacovm (Thu, 13 Sep 2018 10:11:17 GMT):
no
yacovm (Thu, 13 Sep 2018 10:11:27 GMT):
and if it is, then it's wrong.
yacovm (Thu, 13 Sep 2018 10:11:37 GMT):
because the orderer has no use of this information
yacovm (Thu, 13 Sep 2018 10:11:52 GMT):
i.e - if i'm running my orderer in front of a load balancer, a firewall, a proxy, etc. etc.
yacovm (Thu, 13 Sep 2018 10:12:06 GMT):
I need the TLS server cert to have that instead of my own ip/host
yacovm (Thu, 13 Sep 2018 10:12:18 GMT):
I can also, run inside a container, and have an ip that is like 172.20.0.5
yacovm (Thu, 13 Sep 2018 10:12:44 GMT):
let's not make things complex where we can keep them simple ;)
adarshsaraf123 (Thu, 13 Sep 2018 10:13:33 GMT):
Oh okay. I get what you are saying. Then essentially [this specification](https://github.com/hyperledger/fabric/blob/master/sampleconfig/orderer.yaml#L24-#L28) of `ListenAddress` and `ListenPort` does not really help us?
yacovm (Thu, 13 Sep 2018 10:14:44 GMT):
listen address is what you bind to
yacovm (Thu, 13 Sep 2018 10:14:48 GMT):
it can be like 0.0.0.0
adarshsaraf123 (Thu, 13 Sep 2018 10:15:14 GMT):
And then what purpose does the host/ip specification in the `Consenter` struct serve?
yacovm (Thu, 13 Sep 2018 10:15:29 GMT):
that's how other nodes reach you
yacovm (Thu, 13 Sep 2018 10:15:44 GMT):
and i'm saying - there is nothing you can compare it to
adarshsaraf123 (Thu, 13 Sep 2018 10:17:49 GMT):
Ok got it. So I am to summarize, there is a static ip/port that a consenting node has to be reachable at as specified in the `Consenter` but while we spin up the node we cannot really check if the node is reachable at that. And it is the job of the admin to ensure that it is indeed reachable at the specified address. Am I right?
adarshsaraf123 (Thu, 13 Sep 2018 10:17:49 GMT):
Ok got it. So I am to summarize, there _is_ a static ip/port that a consenting node has to be reachable at as specified in the `Consenter` but while we spin up the node we cannot really check if the node is reachable at that. And it is the job of the admin to ensure that it is indeed reachable at the specified address. Am I right?
yacovm (Thu, 13 Sep 2018 10:18:21 GMT):
yes
adarshsaraf123 (Thu, 13 Sep 2018 10:18:50 GMT):
OK thanks. And even the client cert need not be compared against?
yacovm (Thu, 13 Sep 2018 10:19:01 GMT):
let's keep it simple for now
yacovm (Thu, 13 Sep 2018 10:19:05 GMT):
we can always add it
adarshsaraf123 (Thu, 13 Sep 2018 10:19:18 GMT):
Ok sure. Thanks again :)
yacovm (Thu, 13 Sep 2018 10:19:43 GMT):
the way i see it, we need to get a full specification working first
yacovm (Thu, 13 Sep 2018 10:19:48 GMT):
all these misc can always be added later
adarshsaraf123 (Thu, 13 Sep 2018 10:20:04 GMT):
Got it :+1_tone4:
JaydipMakadia (Thu, 13 Sep 2018 13:12:23 GMT):
Has joined the channel.
guoger (Fri, 14 Sep 2018 02:06:40 GMT):
so, we have 4 people working on the same CR stack with mixed dependencies... how do you guys think that we could streamline the review pipeline...
C0rWin (Fri, 14 Sep 2018 08:15:34 GMT):
I think we can continue carefully build on top of this stack upcoming changes making sure to rebase and resolving conflicts
C0rWin (Fri, 14 Sep 2018 08:16:27 GMT):
also I think we need to try to design interfaces which will reduce frictions of conflicts
C0rWin (Fri, 14 Sep 2018 08:16:41 GMT):
for example if we have had separation between chain and raft fsm we might have had less contention
C0rWin (Fri, 14 Sep 2018 08:17:09 GMT):
currently it seems that all 4 of us working on single piece i.e. `chain.go`
guoger (Fri, 14 Sep 2018 08:28:43 GMT):
ha, we might end up working on another single piece still :P and I don't see an obvious line in between, we may frequently change the design of interface. So I proposed to couple them for faster iteration.
guoger (Fri, 14 Sep 2018 08:28:43 GMT):
ha, we might end up working on another single piece still :P. I don't see an obvious line between raft and chain, we may frequently change the design of interface. So I proposed to couple them for faster iteration.
guoger (Fri, 14 Sep 2018 08:29:12 GMT):
we need to take care of this: https://jira.hyperledger.org/browse/FAB-12003 cc @jyellick
kostas (Sat, 15 Sep 2018 19:09:05 GMT):
@guoger: This might be a n00b question, but I don't quite get the concept of a watcher in clockfoundry's fake clock. (And GoDoc/Google are not helping at all.) High-level overview of what it's meant to do?
kostas (Sat, 15 Sep 2018 19:09:15 GMT):
(Looking at your clock CR right now.)
kostas (Sat, 15 Sep 2018 20:33:41 GMT):
Do we have any idea why I'm unable to rebase this one? https://gerrit.hyperledger.org/r/c/26037/
kostas (Sat, 15 Sep 2018 20:34:08 GMT):
I see no option to do so, and cherry-picking it on top of Jay's updated stack and pushing it gives me:
kostas (Sat, 15 Sep 2018 20:34:33 GMT):
```~/Go/src/github.com/hyperledger/fabric/orderer/consensus (fab-11162-jay) $ git lg | head -n4
* 818b91713 - (HEAD -> fab-11162-jay) [FAB-11832] Extract registrar initialization logic (54 seconds ago)
yacovm (Sat, 15 Sep 2018 20:48:57 GMT):
yes, because @guoger really likes his drafts
yacovm (Sat, 15 Sep 2018 20:49:01 GMT):
@kostas
yacovm (Sat, 15 Sep 2018 20:49:10 GMT):
I fixed it... should work now
yacovm (Sat, 15 Sep 2018 20:49:25 GMT):
and the TLS change set is merged
kostas (Sat, 15 Sep 2018 20:49:30 GMT):
Ah, I see. Sorry about that. I know rebasing is a pain.
yacovm (Sat, 15 Sep 2018 20:51:15 GMT):
it's not a pain through gerrit UI
guoger (Sun, 16 Sep 2018 05:01:00 GMT):
:joy: no more draft
guoger (Sun, 16 Sep 2018 05:03:31 GMT):
> the concept of a watcher in clockfoundry's fake clock.
Are you talking about the `WaitForWatcherAndIncrement`? so when a timer is started with fakeclock, it uses a watcher to track it. this method make sure that the timer is actually started before advance the clock, so that we can deterministically assert the result. Does this answer help?
yacovm (Sun, 16 Sep 2018 08:56:38 GMT):
Guys, can we please try to have a single change set chain and not a tree of change sets?
yacovm (Sun, 16 Sep 2018 08:57:11 GMT):
otherwise we're going to merge-conflict-resolution-hell
yacovm (Sun, 16 Sep 2018 10:59:21 GMT):
@guoger can you please put https://gerrit.hyperledger.org/r/#/c/25479/ somewhere inside the change set chain?
yacovm (Sun, 16 Sep 2018 11:00:27 GMT):
without it being in the chain, we can't create channels, because https://gerrit.hyperledger.org/r/#/c/26286/ is not below and not on top of it
yacovm (Sun, 16 Sep 2018 11:02:01 GMT):
I want tomorrow to enable multi-raft e2e_cli
yacovm (Sun, 16 Sep 2018 11:02:22 GMT):
but i can't do that without having a chain with both the ability to create channels, and multi-raft chains ;)
kostas (Sun, 16 Sep 2018 15:46:46 GMT):
@guoger: I'm not quite sure I get you just yet :)
kostas (Sun, 16 Sep 2018 15:46:56 GMT):
For instance: `Eventually(clock.WatcherCount).Should(Equal(2))`
kostas (Sun, 16 Sep 2018 15:47:01 GMT):
I don't get this.
adarshsaraf123 (Sun, 16 Sep 2018 15:58:58 GMT):
@kostas To the extent I understand, and @guoger can correct me if I am wrong, the fake clock works in such a way that each time a `Timer` or a `Ticker` is created, the clock keeps track of them as `Watcher`s. In the case of `Eventually(clock.WatcherCount).Should(Equal(2))` we are expecting two watchers of the fake clock. One is the `Ticker` for ticking the raft node, and the other is a `Timer` for the batch timeout that ought to be started post receiving of a normal block.
kostas (Sun, 16 Sep 2018 16:02:25 GMT):
Ahhh, this makes sense.
kostas (Sun, 16 Sep 2018 16:02:25 GMT):
Ahhh, this makes sense now.
kostas (Sun, 16 Sep 2018 16:02:29 GMT):
Thank you sir.
kostas (Sun, 16 Sep 2018 16:02:29 GMT):
Thank you both.
guoger (Mon, 17 Sep 2018 01:05:42 GMT):
sorry I did a poor job explaining this...
guoger (Mon, 17 Sep 2018 01:06:56 GMT):
thank you @adarshsaraf123
guoger (Mon, 17 Sep 2018 02:16:03 GMT):
@yacovm rebased the stack. "Say hello to Raft OSN" is updated with trivial changes to be rebased
yacovm (Mon, 17 Sep 2018 04:28:59 GMT):
Trivial changes you mean it's a draft now, @guoger ?
guoger (Mon, 17 Sep 2018 04:31:13 GMT):
No... no draft anymore
guoger (Mon, 17 Sep 2018 04:32:34 GMT):
I think just some comments. It was not rebased on latest version of first two CRs in the stack
yacovm (Mon, 17 Sep 2018 05:00:38 GMT):
By the way
yacovm (Mon, 17 Sep 2018 05:01:57 GMT):
@guoger that was a joke 😉
guoger (Mon, 17 Sep 2018 05:03:54 GMT):
lol, for a moment I thought I pushed to draft again :P
yacovm (Mon, 17 Sep 2018 06:11:37 GMT):
@kostas @guoger @adarshsaraf123 @C0rWin - the change set chain is still not properly rebased, I think.... I'm doing a cascading rebase starting from the first CR to figure out where it is split
yacovm (Mon, 17 Sep 2018 06:12:15 GMT):
I'm rebasing via gerrit UI, so please when you update a CR - pull the CR from gerrit before continuing the work, otherwise you'll end up un-rebasing :(
yacovm (Mon, 17 Sep 2018 06:14:32 GMT):
@guoger - this one https://gerrit.hyperledger.org/r/#/c/26180/ needs a manual rebase as it can't be automatically rebased
guoger (Mon, 17 Sep 2018 06:15:18 GMT):
that's after multi-node and can you ignore that for now? i don't know whether that should be after/before Adarsh's work yet...
guoger (Mon, 17 Sep 2018 06:15:40 GMT):
so it's currently a cousin of his work now..
adarshsaraf123 (Mon, 17 Sep 2018 06:15:44 GMT):
@yacovm @C0rWin @guoger @kostas
I was thinking of calling the erstwhile `FSMConfig` just `Options` It will look good in the `configtxyaml` as yacov had pointed out and spells out the meaning also sufficiently clearly. If there is consensus on this then I can proceed with making the change.
guoger (Mon, 17 Sep 2018 06:16:02 GMT):
Options LGTM
yacovm (Mon, 17 Sep 2018 06:17:53 GMT):
@guoger - your change set is only above https://gerrit.hyperledger.org/r/#/c/26038/ but it's not above https://gerrit.hyperledger.org/r/#/c/26287/ : (
yacovm (Mon, 17 Sep 2018 06:17:53 GMT):
@guoger - your change set is only above https://gerrit.hyperledger.org/r/#/c/26038/ but it's not above https://gerrit.hyperledger.org/r/#/c/26287/ :(
yacovm (Mon, 17 Sep 2018 06:18:04 GMT):
do you mind if I try rebasing it on top?
guoger (Mon, 17 Sep 2018 06:22:44 GMT):
go ahead, if it's blocking anything, throw it to the tail. It shouldn't not be in mvp anyway
guoger (Mon, 17 Sep 2018 06:23:03 GMT):
(don't bother to rebase it, just put it to tail. I'll rebase)
yacovm (Mon, 17 Sep 2018 06:52:28 GMT):
so I had to solve conflicts, @guoger
yacovm (Mon, 17 Sep 2018 06:52:40 GMT):
and now the consenter tests aren't passing :/
guoger (Mon, 17 Sep 2018 06:53:08 GMT):
I had similar _'concern'_ as @adarshsaraf123 w.r.t decoupling and my understanding is that it's ok for a leader to fail to write _all_ blocks in its queue into ledger, as long as we guarantee that new leader creates new blocks starting from last _applied_ block. Although, I just wanna point out that it may get a bit more complicated if we want to prevent new normal blocks while a config block is in-flight
guoger (Mon, 17 Sep 2018 06:53:41 GMT):
@yacovm where did you see conflict? nothing should be depending on wal yet, no?
yacovm (Mon, 17 Sep 2018 06:53:50 GMT):
unrelated to WAL
yacovm (Mon, 17 Sep 2018 06:54:31 GMT):
https://gerrit.hyperledger.org/r/#/c/25479/
guoger (Mon, 17 Sep 2018 06:54:33 GMT):
in my local stack, multi-node comes right after 'say hello', and you want to inject 'creation' in between?
yacovm (Mon, 17 Sep 2018 06:54:36 GMT):
the integration tests pass
yacovm (Mon, 17 Sep 2018 06:54:39 GMT):
https://gerrit.hyperledger.org/r/#/c/25479/
kostas (Mon, 17 Sep 2018 06:54:41 GMT):
> I had similar _'concern'_ as @adarshsaraf123 w.r.t decoupling and my understanding is that it's ok for a leader to fail to write _all_ blocks in its queue into ledger, as long as we guarantee that new leader creates new blocks starting from last _applied_ block.
@guoger: Jay, what else could it start creating new blocks on, other than the last applied block?
yacovm (Mon, 17 Sep 2018 06:54:44 GMT):
but the UT don't...
yacovm (Mon, 17 Sep 2018 06:54:51 GMT):
@guoger can you please take a look at https://gerrit.hyperledger.org/r/#/c/25479/ ?
yacovm (Mon, 17 Sep 2018 06:54:56 GMT):
these are your tests :joy:
yacovm (Mon, 17 Sep 2018 06:55:12 GMT):
I want to spearhead into multi-node raft e2e_cli if possible
guoger (Mon, 17 Sep 2018 06:55:35 GMT):
looking
yacovm (Mon, 17 Sep 2018 06:56:05 GMT):
> in my local stack, multi-node comes right after 'say hello', and you want to inject 'creation' in between?
So, I want to put the multi-node right after https://gerrit.hyperledger.org/r/#/c/26297/
yacovm (Mon, 17 Sep 2018 06:56:23 GMT):
the reason is - the https://gerrit.hyperledger.org/r/#/c/26297/ (single node integration tests) is in a way -a checkpoint in the development
yacovm (Mon, 17 Sep 2018 06:56:29 GMT):
that "things work good for a single node"
guoger (Mon, 17 Sep 2018 07:05:57 GMT):
fixed. two lines were missing from rebasing..
guoger (Mon, 17 Sep 2018 07:06:31 GMT):
@yacovm can you give it another try?
yacovm (Mon, 17 Sep 2018 07:09:51 GMT):
yay it works, thanks
guoger (Mon, 17 Sep 2018 07:27:32 GMT):
> what else could it start creating new blocks on, other than the last applied block?
@kostas that's to say, we should discard blocks in queue if leadership is lost, with the same reason for purging blockcutter. And read last applied when elected.
guoger (Mon, 17 Sep 2018 07:27:54 GMT):
I'm not saying it's a problem, just wanna make sure it's taken care of
guoger (Mon, 17 Sep 2018 07:28:35 GMT):
so when we create jira, it may be worth to be added to description, cc @adarshsaraf123
yacovm (Mon, 17 Sep 2018 08:30:01 GMT):
ok it's not working, guys
yacovm (Mon, 17 Sep 2018 08:30:04 GMT):
```
2018-09-17 08:28:14.094 UTC [orderer/common/cluster] handleSubmit -> WARN 20f Handling of Propose() from 192.168.192.7:56099 failed: badly formatted message, cannot extract channel
github.com/hyperledger/fabric/orderer/common/cluster.(*Comm).requestContext
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/cluster/comm.go:124
github.com/hyperledger/fabric/orderer/common/cluster.(*Comm).DispatchSubmit
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/cluster/comm.go:102
github.com/hyperledger/fabric/orderer/common/cluster.(*Service).handleSubmit
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/cluster/service.go:82
github.com/hyperledger/fabric/orderer/common/cluster.(*Service).Submit
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/cluster/service.go:61
github.com/hyperledger/fabric/protos/orderer._Cluster_Submit_Handler
/opt/gopath/src/github.com/hyperledger/fabric/protos/orderer/cluster.pb.go:312
github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).processStreamingRPC
/opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:1160
github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).handleStream
/opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:1253
github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1
/opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:680
runtime.goexit
/opt/go/src/runtime/asm_amd64.s:2361
github.com/hyperledger/fabric/orderer/common/cluster.(*Comm).DispatchSubmit
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/cluster/comm.go:104
github.com/hyperledger/fabric/orderer/common/cluster.(*Service).handleSubmit
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/cluster/service.go:82
github.com/hyperledger/fabric/orderer/common/cluster.(*Service).Submit
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/cluster/service.go:61
github.com/hyperledger/fabric/protos/orderer._Cluster_Submit_Handler
/opt/gopath/src/github.com/hyperledger/fabric/protos/orderer/cluster.pb.go:312
github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).processStreamingRPC
/opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:1160
github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).handleStream
/opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:1253
github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1
```
yacovm (Mon, 17 Sep 2018 08:30:17 GMT):
but hey at least the communication is working :)
yacovm (Mon, 17 Sep 2018 08:30:42 GMT):
going now to a meeting and will investigate after, but looks like someone didn't put a channel tag maybe
C0rWin (Mon, 17 Sep 2018 08:33:15 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=dJZFWeifTsGrTkGBw) @adarshsaraf123 I'm good with it
yacovm (Mon, 17 Sep 2018 14:43:37 GMT):
attention all - I have rebased today the entire CR stack, please take care in pushing change sets :)
yacovm (Mon, 17 Sep 2018 14:43:37 GMT):
attention all - I have rebased today the entire CR stack, please take care in pushing change sets :)
@kostas @adarshsaraf123 @C0rWin
yacovm (Mon, 17 Sep 2018 14:43:37 GMT):
attention all - I have rebased today the entire CR stack, please take care in pushing change sets :)
@kostas @adarshsaraf123 @C0rWin
yacovm (Mon, 17 Sep 2018 14:43:37 GMT):
attention all - I have rebased today the entire CR stack, to fix something in a CR in between, and also to pick up latest code changes from master. please take care in pushing change sets :)
@kostas @adarshsaraf123 @C0rWin
yacovm (Mon, 17 Sep 2018 14:43:48 GMT):
@guoger I have updated slightly your change set for multi-node raft
yacovm (Mon, 17 Sep 2018 14:43:59 GMT):
Since rocket was down I was unable to contact you to fix it
yacovm (Mon, 17 Sep 2018 14:44:10 GMT):
but basically - i added the channel ID to the `chain` struct
yacovm (Mon, 17 Sep 2018 14:44:21 GMT):
and added the channel ID to the `Submit()` invocations to the comm layer
yacovm (Mon, 17 Sep 2018 14:44:30 GMT):
otherwise - the comm layer on the other side rejects the messages
Jgnuid (Mon, 17 Sep 2018 15:14:07 GMT):
Has joined the channel.
kostas (Mon, 17 Sep 2018 15:19:29 GMT):
The Gods have blessed us by taking RC down. Praise be. It was good while it lasted.
snakejerusalem (Mon, 17 Sep 2018 15:36:06 GMT):
Has joined the channel.
snakejerusalem (Mon, 17 Sep 2018 15:41:05 GMT):
Greetings. I am not sure if this is the correct channel, so apologies in advance. I am working with my own Fabric v1.2 fork, and I currently need to fetch the orderer type from within this method: `https://github.com/hyperledger/fabric/blob/release-1.2/core/committer/txvalidator/validator.go#L264`. I've seen that the TxValidator interface provides a Support and Vscc attribute. Is it possible to fetch the orderer type using any of these methods?
jyellick (Mon, 17 Sep 2018 17:31:36 GMT):
@snakejerusalem The orderer type can be retrieved via the channel configuration. You could likely extend the `Support` interface to also return the OrdererConfig in addition to the ApplicationConfig data.
jyellick (Mon, 17 Sep 2018 17:32:19 GMT):
However, as a design principle, the peers should generally be agnostic to the orderer implementation type. The only piece that changes is typically the 'block validation policy' which determines who and how many valid signatures a block must have
yacovm (Mon, 17 Sep 2018 17:45:42 GMT):
snake, here is sample code that does what you need
```
func ordererType(genesisBlock *cb.Block) string {
if genesisBlock.Data == nil || len(genesisBlock.Data.Data) == 0 {
logger.Fatalf("Empty genesis block")
}
env := &cb.Envelope{}
if err := proto.Unmarshal(genesisBlock.Data.Data[0], env); err != nil {
logger.Fatalf("Failed to unmarshal the genesis block's envelope: %v", err)
}
bundle, err := channelconfig.NewBundleFromEnvelope(env)
if err != nil {
logger.Fatalf("Failed creating bundle from the genesis block: %v", err)
}
ordConf, exists := bundle.OrdererConfig()
if !exists {
logger.Fatalf("Orderer config doesn't exist in bundle derived from genesis block")
}
return ordConf.ConsensusType()
}
`````
snakejerusalem (Mon, 17 Sep 2018 17:46:48 GMT):
Thanks, @jyellick . But unfortunately, I have an issue that really forces me to do one of two things: a) assume that I always use the same orderer type, or b) break that principle so that I know I can do some thing different that my orderer needs
snakejerusalem (Mon, 17 Sep 2018 17:47:04 GMT):
@yacovm thank you very much!
snakejerusalem (Mon, 17 Sep 2018 17:54:24 GMT):
btw, the issue I have is related to the fact that configuration envelopes need to be signed by the orderer. However, from what I understand of Fabric, I do not see why they need to be. When a configuration envelope is created, it is placed in block all to it self, and blocks are always signed by the orderer. So if that configuration block is signed, does the configuration envelope inside it really needs t be signed as well?
jyellick (Mon, 17 Sep 2018 17:58:34 GMT):
You may think of it as more of an audit trail. Some orderer receives a 'config update', from a client. That orderer validates that update, and produces a new config based on that update. Then, the config is ordered, and each orderer optionally validates (depending on consensus type) and commits the config. When the peers receive the config, they similarly validate the update, before committing the block. The config should _always_ validate, because the orderers should have validated it before committing. However, if the config does not validate, having evidence (an ID + signature) of which orderer generated the config is valuable.
jyellick (Mon, 17 Sep 2018 17:58:34 GMT):
You may think of it as more of an audit trail. Some orderer receives a 'config update', from a client. That orderer validates that update, and produces a new config based on that update. Then, the config is ordered, and each orderer optionally re-validates (depending on consensus type) and commits the config. When the peers receive the config, they similarly validate the update, before committing the block. The config should _always_ validate, because the orderers should have validated it before committing. However, if the config does not validate, having evidence (an ID + signature) of which orderer generated the config is valuable.
snakejerusalem (Mon, 17 Sep 2018 18:05:47 GMT):
But both the config and the block are signe by the orderers. Since the config is added to a block comprised by only that config, aren't one fo the signatures redundant?
snakejerusalem (Mon, 17 Sep 2018 18:05:47 GMT):
But both the config and the block are signed by the orderers. Since the config is added to a block comprised by only that config, isn't one of the signatures redundant?
snakejerusalem (Mon, 17 Sep 2018 18:05:47 GMT):
But both the config and the block are signed by the orderers. Since the config is added to a block comprised by only that config, isn't one of the signatures redundant? Don't both signatures serve to validate the same thing?
snakejerusalem (Mon, 17 Sep 2018 18:09:07 GMT):
And since all blocks need to be signed, isn't it theoretically safe to leave the config unsigned?
jyellick (Mon, 17 Sep 2018 18:18:28 GMT):
Depends on your assumptions. It is a nice feature of fabric, that every envelope on the blockchain is signed by the entity which created it. It leaves a nice audit trail. Because in the special case of config transactions, the orderers do some additional validation, you could argue that the signature is redundant. However, for instance in the CFT case like with Kafka, the other orderers do not generally revalidate the config, they simply commit it, so the signature on the block is attesting to something slightly different.
snakejerusalem (Mon, 17 Sep 2018 18:35:19 GMT):
yes, but once you start working with the BFT case, this is a problem. In CFT one signature is enough, but in BFT you need a Byzantine majority of signatures. Unfortunately the envelopes only support a single signature, whereas the blocks support multiple signatures.
jyellick (Mon, 17 Sep 2018 18:36:04 GMT):
In the BFT case, the signature over the config would generally be that of the node which generated it (usually the leader at the time). Though the f+1 signatures on the block would attest to the config's validity.
snakejerusalem (Mon, 17 Sep 2018 18:37:41 GMT):
but why would a single signature from a leader (that could very well be malicious) be enough for safety?
snakejerusalem (Mon, 17 Sep 2018 18:42:15 GMT):
I don't see how you could preserve safety in the BFT case having the config envelope be signed by a single orderer, be it the leader or other process.
jyellick (Mon, 17 Sep 2018 19:07:25 GMT):
In the BFT case, it is not for safety, it's for audit.
jyellick (Mon, 17 Sep 2018 19:09:22 GMT):
As you say, if all of the BFT orderers validate the config before agreeing to commit it, there is no safety concern. However, it is nice to be able to associate any generated transaction content with a particular identity.
snakejerusalem (Mon, 17 Sep 2018 19:15:59 GMT):
ok, gotcha.
snakejerusalem (Mon, 17 Sep 2018 19:16:35 GMT):
Also, one other thing, I am having trouble finding in the source code the place where the block signatures are verified. Where is that done?
jyellick (Mon, 17 Sep 2018 19:20:39 GMT):
https://github.com/hyperledger/fabric/blob/77c3aa6ce5b0cfba93bfda009095886dbcadff91/peer/gossip/mcs.go#L117-L196
snakejerusalem (Mon, 17 Sep 2018 19:21:46 GMT):
thank you!
snakejerusalem (Mon, 17 Sep 2018 20:10:39 GMT):
Just one last thing. Fabric generates a second block signature here: https://github.com/hyperledger/fabric/blob/3ec2839de4d0d754b1eaf01bd18b389e3edd2086/orderer/common/multichannel/blockwriter.go#L195 . But where in the code is this signature validated? I found some places where the method `GetLastConfigIndexFromBlock` is used to fetch the value encoded in the metadata, but where is the respective signature validated?
yacovm (Mon, 17 Sep 2018 20:27:27 GMT):
so, the truth is - it is not used....
yacovm (Mon, 17 Sep 2018 20:27:50 GMT):
it is only used when it has already been committed
yacovm (Mon, 17 Sep 2018 20:28:04 GMT):
but - it is never used on a block that is pending to be committed, or in-flight
yacovm (Mon, 17 Sep 2018 20:28:27 GMT):
and since you committed the block, you also checked its height.
yacovm (Mon, 17 Sep 2018 20:29:00 GMT):
and i have to ask what's the deal with your name?
yacovm (Mon, 17 Sep 2018 20:29:00 GMT):
and i have to ask what's the deal with your name? (what does it mean?)
snakejerusalem (Mon, 17 Sep 2018 20:32:46 GMT):
Ok, thank you! a for my username, it is a reference to a comic book called "transmetropolitan". The protagonist is called "Spider Jerusalem", but I changed it slightly so that I can always
snakejerusalem (Mon, 17 Sep 2018 20:32:46 GMT):
Ok, thank you! a for my username, it is a reference to a comic book called "transmetropolitan". The protagonist is called "Spider Jerusalem", but I changed it slightly so that I can always fetch an unused username anywhere I sign up to.
yacovm (Mon, 17 Sep 2018 20:35:36 GMT):
> Spider's weapon of choice for most of the series is a "bowel disruptor," which causes instant and painful loss of bowel control, with settings that allow him to vary the level of pain and discomfort, ranging from simple loose, watery diarrhea to complete rectal prolapse.
Seems like a nice man, overall.
snakejerusalem (Mon, 17 Sep 2018 20:37:35 GMT):
Definitely a model citizen ^^
yacovm (Tue, 18 Sep 2018 07:44:42 GMT):
@guoger @adarshsaraf123 where did you change the FSMConfig to Options ?
yacovm (Tue, 18 Sep 2018 07:44:44 GMT):
in what change set?
yacovm (Tue, 18 Sep 2018 07:45:31 GMT):
ah found it https://gerrit.hyperledger.org/r/#/c/26178/
huikang (Wed, 19 Sep 2018 04:47:48 GMT):
Hi, I have a noob question about these two lines (https://github.com/hyperledger/fabric/blob/e6bdd232138c7e22ccb373587956ee6f5c18725a/orderer/consensus/etcdraft/chain.go#L190-L191). My understanding is that if the submitted message `msg.LastValidationSeq ` is behind current channel's config seq, it calls ProcessNormalMsg(msg.Content). However, when I look at the function ProcessNormalMsg, it seems not check the configuration seq and returns nil, which means this msg will be sent to the cutter. Will this cause any conflict if a tx with outdated config seq is sent to the current order? Thanks.
adarshsaraf123 (Wed, 19 Sep 2018 05:03:05 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=9rnC4swCGz7YYNYFy) @huikang Not sure which `ProcessNormalMsg` implementation you have looked at. Check [this one](https://github.com/hyperledger/fabric/blob/e6bdd232138c7e22ccb373587956ee6f5c18725a/orderer/common/msgprocessor/standardchannel.go#L77-#L83) out.
huikang (Wed, 19 Sep 2018 13:59:50 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=QSbddsrz4qgXnrLLb) @adarshsaraf123 Yes, I was looking at this one. It gets the current seq and apply a filter. However, the code in chain.go still has the msg.LastValidationSeq < seq. I am trying to understand in which case, the tx msg is considered a bad one if msg.LastValidationSeq < seq.
adarshsaraf123 (Wed, 19 Sep 2018 14:04:24 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=2N5QwWFjdgJGwxapH) @huikang Oh got you now.. Let's say, hypothetically, that the `AbsoluteMaxBytes` was earlier 5MB and tx A comes in of size 2MB. Currently A is a valid tx. Now let us assume that a config block is created with changes the `AbsoluteMaxBytes` to 1MB. Then this tx will no longer be valid and would be rejected at `ProcessNormalMsg`.
adarshsaraf123 (Wed, 19 Sep 2018 14:04:24 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=2N5QwWFjdgJGwxapH) @huikang Oh got you now.. Let's say, hypothetically, that the `AbsoluteMaxBytes` was earlier 5MB and tx A comes in of size 2MB. Currently A is a valid tx. Now let us assume that a config block is created which changes the `AbsoluteMaxBytes` to 1MB. Then this tx will no longer be valid and would be rejected at `ProcessNormalMsg`.
adarshsaraf123 (Wed, 19 Sep 2018 14:05:24 GMT):
A similar example could also be created with signature invalidation due to an `Org` update in the config block.
huikang (Wed, 19 Sep 2018 14:08:20 GMT):
@adarshsaraf123 thanks for providing the example. That's what I am looking for. However, where can I find in the call path of ProcessNormalMsg that rejects the tx (is it part of the filter)?
adarshsaraf123 (Wed, 19 Sep 2018 14:08:53 GMT):
Yes it is part of the filters.
huikang (Wed, 19 Sep 2018 14:13:28 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=gCb8CvwzmpXDEBCD8) @adarshsaraf123 Looks like the filters are very simple (https://github.com/hyperledger/fabric/blob/e6bdd232138c7e22ccb373587956ee6f5c18725a/orderer/common/msgprocessor/filter.go). My assumption is that the support chain will create some specific rules to check those configurations. Is that correct?
adarshsaraf123 (Wed, 19 Sep 2018 14:17:18 GMT):
[These](https://github.com/hyperledger/fabric/blob/master/orderer/common/msgprocessor/standardchannel.go#L47-#L59) are the filters for the standard channels.
adarshsaraf123 (Wed, 19 Sep 2018 14:18:03 GMT):
[These](https://github.com/hyperledger/fabric/blob/master/orderer/common/msgprocessor/systemchannel.go#L43-#L56) are the filters for the system channels.
adarshsaraf123 (Wed, 19 Sep 2018 14:18:42 GMT):
@huikang ^^
huikang (Wed, 19 Sep 2018 14:53:20 GMT):
Got you. I see the SizeFilter as you described in the example. Thanks!
guoger (Thu, 20 Sep 2018 05:54:06 GMT):
theoretically speaking, can we use Ethereum as OSN?
Shyam_Pratap_Singh (Thu, 20 Sep 2018 06:25:55 GMT):
Has joined the channel.
huikang (Thu, 20 Sep 2018 13:44:28 GMT):
@guoger, I think so as long as the output blocks are the same order for all peers.
guoger (Fri, 21 Sep 2018 07:36:24 GMT):
I'm not sure if `etcd/contrib/raftexample` actually handles snapshotting correctly. can someone help me verify if this failure could be reproduced?
_I'm on master branch with commit `1a3be73`_
change [defaultSnapshotCount](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/raft.go#L74) to smaller number, i.e. 10
*Failure case A:*
1) start a fresh raftexample cluster per [readme](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/README.md#fault-tolerance)
2) run `while curl -L http://127.0.0.1:12380/my-key -XPUT -d foo; do :; done`
3) it panics at second snapshot with
```
15:19:14 raftexample3 | panic: requested index is unavailable due to compaction
15:19:14 raftexample3 | goroutine 111 [running]:
15:19:14 raftexample3 | main.(*raftNode).maybeTriggerSnapshot(0xc4201be000)
15:19:14 raftexample3 | /Users/guoger/workspace/go-project/src/go.etcd.io/etcd/contrib/raftexample/raft.go:373 +0x405
15:19:14 raftexample3 | main.(*raftNode).serveChannels(0xc4201be000)
15:19:14 raftexample3 | /Users/guoger/workspace/go-project/src/go.etcd.io/etcd/contrib/raftexample/raft.go:442 +0x548
15:19:14 raftexample3 | created by main.(*raftNode).startRaft
15:19:14 raftexample3 | /Users/guoger/workspace/go-project/src/go.etcd.io/etcd/contrib/raftexample/raft.go:313 +0x5d3
```
*Failure case B:*
1) start a fresh raftexample cluster per [readme](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/README.md#fault-tolerance)
2) repeatedly insert key till first compact (raft log should say `compacted log at index X`)
3) kill cluster
4) restart with goreman, and all nodes *stuck* at `loading snapshot at term X and index Y`. (unable to insert new key, service is not started)
guoger (Fri, 21 Sep 2018 07:36:24 GMT):
I'm not sure if `etcd/contrib/raftexample` actually handles snapshotting correctly. can someone help me verify if this failure could be reproduced?
_I'm on master branch with commit `1a3be73` _
change [defaultSnapshotCount](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/raft.go#L74) to smaller number, i.e. 10
*Failure case A:*
1) start a fresh raftexample cluster per [readme](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/README.md#fault-tolerance)
2) run `while curl -L http://127.0.0.1:12380/my-key -XPUT -d foo; do :; done`
3) it panics at second snapshot with
```
15:19:14 raftexample3 | panic: requested index is unavailable due to compaction
15:19:14 raftexample3 | goroutine 111 [running]:
15:19:14 raftexample3 | main.(*raftNode).maybeTriggerSnapshot(0xc4201be000)
15:19:14 raftexample3 | /Users/guoger/workspace/go-project/src/go.etcd.io/etcd/contrib/raftexample/raft.go:373 +0x405
15:19:14 raftexample3 | main.(*raftNode).serveChannels(0xc4201be000)
15:19:14 raftexample3 | /Users/guoger/workspace/go-project/src/go.etcd.io/etcd/contrib/raftexample/raft.go:442 +0x548
15:19:14 raftexample3 | created by main.(*raftNode).startRaft
15:19:14 raftexample3 | /Users/guoger/workspace/go-project/src/go.etcd.io/etcd/contrib/raftexample/raft.go:313 +0x5d3
```
*Failure case B:*
1) start a fresh raftexample cluster per [readme](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/README.md#fault-tolerance)
2) repeatedly insert key till first compact (raft log should say `compacted log at index X`)
3) kill cluster
4) restart with goreman, and all nodes *stuck* at `loading snapshot at term X and index Y`. (unable to insert new key, service is not started)
guoger (Fri, 21 Sep 2018 07:36:24 GMT):
I'm not sure if `etcd/contrib/raftexample` actually handles snapshotting correctly. can someone help me verify if this failure could be reproduced?
_I'm on master branch with commit `1a3be73` _
change [defaultSnapshotCount](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/raft.go#L74) to smaller number, i.e. 10
*Failure case A:*
1) start a fresh raftexample cluster per [readme](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/README.md#fault-tolerance)
2) run `while curl -L http://127.0.0.1:12380/my-key -XPUT -d foo; do :; done`
3) it panics (fairly quick) at second snapshot with
```
15:19:14 raftexample3 | panic: requested index is unavailable due to compaction
15:19:14 raftexample3 | goroutine 111 [running]:
15:19:14 raftexample3 | main.(*raftNode).maybeTriggerSnapshot(0xc4201be000)
15:19:14 raftexample3 | /Users/guoger/workspace/go-project/src/go.etcd.io/etcd/contrib/raftexample/raft.go:373 +0x405
15:19:14 raftexample3 | main.(*raftNode).serveChannels(0xc4201be000)
15:19:14 raftexample3 | /Users/guoger/workspace/go-project/src/go.etcd.io/etcd/contrib/raftexample/raft.go:442 +0x548
15:19:14 raftexample3 | created by main.(*raftNode).startRaft
15:19:14 raftexample3 | /Users/guoger/workspace/go-project/src/go.etcd.io/etcd/contrib/raftexample/raft.go:313 +0x5d3
```
*Failure case B:*
1) start a fresh raftexample cluster per [readme](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/README.md#fault-tolerance)
2) repeatedly insert key till first compact (raft log should say `compacted log at index X`)
3) kill cluster
4) restart with goreman, and all nodes *stuck* at `loading snapshot at term X and index Y`. (unable to insert new key, service is not started)
guoger (Fri, 21 Sep 2018 07:36:24 GMT):
I'm not sure if `etcd/contrib/raftexample` actually handles snapshotting correctly. can someone help me verify if this failure could be reproduced?
_I'm on master branch with commit `1a3be73` _
change [defaultSnapshotCount](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/raft.go#L74) to smaller number, i.e. 10
~*Failure case A:*~ never mind this case, you actually need to change both [defaultSnapshotCount](https://github.com/etcd-io/etcd/blob/2cf47366216a50826aab7c37784eba0c1ced3ee1/contrib/raftexample/raft.go#L74) and [snapshotCatchUpEntriesN](https://github.com/etcd-io/etcd/blob/2cf47366216a50826aab7c37784eba0c1ced3ee1/contrib/raftexample/raft.go#L348)
1) start a fresh raftexample cluster per [readme](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/README.md#fault-tolerance)
2) run `while curl -L http://127.0.0.1:12380/my-key -XPUT -d foo; do :; done`
3) it panics (fairly quick) at second snapshot with
```
15:19:14 raftexample3 | panic: requested index is unavailable due to compaction
15:19:14 raftexample3 | goroutine 111 [running]:
15:19:14 raftexample3 | main.(*raftNode).maybeTriggerSnapshot(0xc4201be000)
15:19:14 raftexample3 | /Users/guoger/workspace/go-project/src/go.etcd.io/etcd/contrib/raftexample/raft.go:373 +0x405
15:19:14 raftexample3 | main.(*raftNode).serveChannels(0xc4201be000)
15:19:14 raftexample3 | /Users/guoger/workspace/go-project/src/go.etcd.io/etcd/contrib/raftexample/raft.go:442 +0x548
15:19:14 raftexample3 | created by main.(*raftNode).startRaft
15:19:14 raftexample3 | /Users/guoger/workspace/go-project/src/go.etcd.io/etcd/contrib/raftexample/raft.go:313 +0x5d3
```
*Failure case B:*
1) start a fresh raftexample cluster per [readme](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/README.md#fault-tolerance)
2) repeatedly insert key till first compact (raft log should say `compacted log at index X`)
3) kill cluster
4) restart with goreman, and all nodes *stuck* at `loading snapshot at term X and index Y`. (unable to insert new key, service is not started)
guoger (Fri, 21 Sep 2018 07:36:24 GMT):
I'm not sure if `etcd/contrib/raftexample` actually handles snapshotting correctly. can someone help me verify if this failure could be reproduced?
_I'm on master branch with commit `1a3be73` _
change [defaultSnapshotCount](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/raft.go#L74) to smaller number, i.e. 10
~*Failure case A:*~ never mind this case, you actually need to change both [defaultSnapshotCount](https://github.com/etcd-io/etcd/blob/2cf47366216a50826aab7c37784eba0c1ced3ee1/contrib/raftexample/raft.go#L74) and [snapshotCatchUpEntriesN](https://github.com/etcd-io/etcd/blob/2cf47366216a50826aab7c37784eba0c1ced3ee1/contrib/raftexample/raft.go#L348)
1) start a fresh raftexample cluster per [readme](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/README.md#fault-tolerance)
2) run `while curl -L http://127.0.0.1:12380/my-key -XPUT -d foo; do :; done`
3) it panics (fairly quick) at second snapshot with
```
15:19:14 raftexample3 | panic: requested index is unavailable due to compaction
15:19:14 raftexample3 | goroutine 111 [running]:
15:19:14 raftexample3 | main.(*raftNode).maybeTriggerSnapshot(0xc4201be000)
15:19:14 raftexample3 | /Users/guoger/workspace/go-project/src/go.etcd.io/etcd/contrib/raftexample/raft.go:373 +0x405
15:19:14 raftexample3 | main.(*raftNode).serveChannels(0xc4201be000)
15:19:14 raftexample3 | /Users/guoger/workspace/go-project/src/go.etcd.io/etcd/contrib/raftexample/raft.go:442 +0x548
15:19:14 raftexample3 | created by main.(*raftNode).startRaft
15:19:14 raftexample3 | /Users/guoger/workspace/go-project/src/go.etcd.io/etcd/contrib/raftexample/raft.go:313 +0x5d3
```
*Failure case B:* (filed and [issue](https://github.com/etcd-io/etcd/issues/10118))
1) start a fresh raftexample cluster per [readme](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/README.md#fault-tolerance)
2) repeatedly insert key till first compact (raft log should say `compacted log at index X`)
3) kill cluster
4) restart with goreman, and all nodes *stuck* at `loading snapshot at term X and index Y`. (unable to insert new key, service is not started)
guoger (Fri, 21 Sep 2018 07:36:24 GMT):
I'm not sure if `etcd/contrib/raftexample` actually handles snapshotting correctly. can someone help me verify if this failure could be reproduced?
_I'm on master branch with commit `1a3be73` _
change [defaultSnapshotCount](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/raft.go#L74) to smaller number, i.e. 10
~*Failure case A:*~ _ignore this case_, you actually need to change both [defaultSnapshotCount](https://github.com/etcd-io/etcd/blob/2cf47366216a50826aab7c37784eba0c1ced3ee1/contrib/raftexample/raft.go#L74) and [snapshotCatchUpEntriesN](https://github.com/etcd-io/etcd/blob/2cf47366216a50826aab7c37784eba0c1ced3ee1/contrib/raftexample/raft.go#L348)
1) start a fresh raftexample cluster per [readme](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/README.md#fault-tolerance)
2) run `while curl -L http://127.0.0.1:12380/my-key -XPUT -d foo; do :; done`
3) it panics (fairly quick) at second snapshot with
```
15:19:14 raftexample3 | panic: requested index is unavailable due to compaction
15:19:14 raftexample3 | goroutine 111 [running]:
15:19:14 raftexample3 | main.(*raftNode).maybeTriggerSnapshot(0xc4201be000)
15:19:14 raftexample3 | /Users/guoger/workspace/go-project/src/go.etcd.io/etcd/contrib/raftexample/raft.go:373 +0x405
15:19:14 raftexample3 | main.(*raftNode).serveChannels(0xc4201be000)
15:19:14 raftexample3 | /Users/guoger/workspace/go-project/src/go.etcd.io/etcd/contrib/raftexample/raft.go:442 +0x548
15:19:14 raftexample3 | created by main.(*raftNode).startRaft
15:19:14 raftexample3 | /Users/guoger/workspace/go-project/src/go.etcd.io/etcd/contrib/raftexample/raft.go:313 +0x5d3
```
*Failure case B:* (filed and [issue](https://github.com/etcd-io/etcd/issues/10118))
1) start a fresh raftexample cluster per [readme](https://github.com/etcd-io/etcd/blob/master/contrib/raftexample/README.md#fault-tolerance)
2) repeatedly insert key till first compact (raft log should say `compacted log at index X`)
3) kill cluster
4) restart with goreman, and all nodes *stuck* at `loading snapshot at term X and index Y`. (unable to insert new key, service is not started)
adarshsaraf123 (Tue, 25 Sep 2018 09:47:02 GMT):
@kostas @guoger @yacovm @C0rWin Have pushed the multi-node etcd raft integration test CR: https://gerrit.hyperledger.org/r/#/c/26547/
Currently I have assigned it the JIRA number FAB-11161 since @kostas had mentioned that the integration tests belong to the parent story. Can be changed as necessary.
adarshsaraf123 (Tue, 25 Sep 2018 09:47:02 GMT):
@kostas @guoger @yacovm @C0rWin Have pushed the multi-node etcd raft integration test CR: https://gerrit.hyperledger.org/r/#/c/26547/
Currently I have assigned it the JIRA number [FAB-11161](https://jira.hyperledger.org/browse/FAB-11161) since @kostas had mentioned that the integration tests belong to the parent story. Can be changed as necessary.
adarshsaraf123 (Tue, 25 Sep 2018 09:49:42 GMT):
@yacovm Despite my comment on [your CR](https://gerrit.hyperledger.org/r/#/c/26297/) I have not changed the ClusterMembers function in `integration/nwo/network.go` since it was comfortable to use it as such. I figured this out later :sweat_smile:
yacovm (Tue, 25 Sep 2018 09:56:04 GMT):
Cool.
yacovm (Tue, 25 Sep 2018 09:56:20 GMT):
I think it needs to be extended to test failure scenarios of the leader, etc.
yacovm (Tue, 25 Sep 2018 09:56:56 GMT):
you can know who is the leader by parsing the certificate from the signature of the latest block and then figuring out what is the subject name.
guoger (Tue, 25 Sep 2018 10:10:06 GMT):
@yacovm aren't you supposed to be on vacation? :P
yacovm (Tue, 25 Sep 2018 10:12:05 GMT):
i am - that's why I'm not doing it myself
yacovm (Tue, 25 Sep 2018 10:12:14 GMT):
I get phone notifications
yacovm (Tue, 25 Sep 2018 10:12:19 GMT):
(if i'm tagged)
adarshsaraf123 (Tue, 25 Sep 2018 10:59:28 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=kC9fdHSFGJZvLvCiY) @yacovm I thought that is anyways taken care of by the unit tests at the etcdraft level.
yacovm (Tue, 25 Sep 2018 11:12:58 GMT):
We need integration scenarios that test failover of nodes, leader recovery, etc. etc.
yacovm (Tue, 25 Sep 2018 11:13:27 GMT):
the UT aren't enough IMO... @guoger @C0rWin and @kostas can chime in too
yacovm (Tue, 25 Sep 2018 11:13:38 GMT):
also @jyellick :)
adarshsaraf123 (Tue, 25 Sep 2018 11:34:00 GMT):
While we wait for the others to chime in, I will present my point of view. I think that the etcdraft cluster can be isolated as a black box once we unit-test the said failures. That this black box indeed works in tandem with the rest of the system is then being tested by the pushed CR. This in my opinion does provide a comprehensive test. Of course I could be missing some details here.
yacovm (Tue, 25 Sep 2018 14:11:22 GMT):
@guoger 's tests use a mock communication layer. How do you test that you don't run into a distributed deadlock because due to something untested in the communication unit tests? The communication unit tests can't mimick the usage pattern of etcdraft's raft.Node, therefore we need to test them together.
yacovm (Tue, 25 Sep 2018 14:12:02 GMT):
also we need to make sure that reconfiguration and addition of new nodes works
jyellick (Tue, 25 Sep 2018 14:22:35 GMT):
My take: We should definitely have some integration tests which verify the interaction between the etcd/raft code and the Fabric code. This should include a green path, and a common failure or two. We certainly don't need or want to re-test etcd/raft via integration tests, but getting a base level of comfort via tests that the pieces do fit together and work seems like a good thing to me.
guoger (Wed, 26 Sep 2018 01:45:01 GMT):
integration.png
guoger (Wed, 26 Sep 2018 01:45:06 GMT):
I tend to agree with @jyellick
adarshsaraf123 (Wed, 26 Sep 2018 05:44:56 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=wq5FLMXwc9rYiSuNk) @jyellick With the [CR pushed](https://gerrit.hyperledger.org/r/#/c/26547/) earlier, we have got the green path covered. I will come up with some failure scenarios at a system level. I am assuming that these failures will have to be something triggered due to the way the orderer interacts with the rest of the system.
adarshsaraf123 (Wed, 26 Sep 2018 05:50:22 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=EYrYx2xbJBqzHBjhg) @yacovm True. These are definitely necessary and, IMO, these tests should be isolated within the orderer codebase since the rest of the system should only see a reliable orderer. I think this also helps keep the orderer completely independent such that someone can just pull the orderer/ subdirectory and work with it for any project. For this to work, I think all the tests for the orderer behaviour per se should be within the orderer codebase.
adarshsaraf123 (Fri, 28 Sep 2018 08:23:04 GMT):
@kostas @yacovm @guoger @C0rWin I am working on the decoupling of `CreateNextBlock` and `WriteBlock`. Currently, [WriteBlock](https://github.com/hyperledger/fabric/blob/master/orderer/common/multichannel/blockwriter.go#L149-#L157) launches a goroutine for `commitBlock`. Now, if there are errors in committing the block like unexpected block number, etc. then `commitBlock` panics. I would like to test this panic of `commitBlock` but I am making the call to only `WriteBlock` within `assert.Panics`. Since the `commitBlock` is in a separate goroutine, `assert.Panics` cannot catch this panic. Does anybody have any idea on how this can be done?
Bartb0 (Tue, 02 Oct 2018 10:56:14 GMT):
Has joined the channel.
kostas (Wed, 03 Oct 2018 19:07:35 GMT):
Hi everyone. I'm back from vacation and in catch-up mode. I intend to go over all the messages I missed here during the past couple of weeks, but in the meantime can I rebase those Raft-related CRs in "Merge Conflict"? Or is there a specific reason these are left untouched?
guoger (Wed, 03 Oct 2018 19:29:55 GMT):
Hey @kostas I'm still on vacation till Oct 6. For wal, I'm rebasing them cuz it conflicts with some of Adarsh's CRs. For multi-node CR, feel free to rebase.
kostas (Wed, 03 Oct 2018 19:30:26 GMT):
Excellent. Thanks for letting me know; we'll see you next week.
kostas (Fri, 05 Oct 2018 14:12:08 GMT):
Just to make sure I'm not losing my mind: is the latest master consistently failing on unit tests for everyone else?
kostas (Fri, 05 Oct 2018 14:12:29 GMT):
The culprit seems to be `github.com/hyperledger/fabric/core/chaincode`
qiangqinqq (Sat, 06 Oct 2018 07:37:06 GMT):
Has joined the channel.
yacovm (Sun, 07 Oct 2018 14:33:48 GMT):
@guoger @adarshsaraf123 I rebased the chain up until https://gerrit.hyperledger.org/r/#/c/26038/, can you guys please continue the rebasing further?
yacovm (Sun, 07 Oct 2018 14:33:56 GMT):
we need to get these merge conflicts solved :)
yacovm (Sun, 07 Oct 2018 14:34:21 GMT):
next in line are https://gerrit.hyperledger.org/r/#/c/26104/ and https://gerrit.hyperledger.org/r/#/c/26178/ @adarshsaraf123 - that's yours
yacovm (Sun, 07 Oct 2018 14:34:33 GMT):
and I don't want to break anything while solving merge conflicts...
adarshsaraf123 (Mon, 08 Oct 2018 05:28:57 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=uWwtmzhSzYfp7qYe9) @yacovm On it :)
yacovm (Mon, 08 Oct 2018 06:13:26 GMT):
@adarshsaraf123 thank you!
guoger (Mon, 08 Oct 2018 08:54:41 GMT):
are we still targeting 1.4? or it's changed to 2.0?
adarshsaraf123 (Mon, 08 Oct 2018 09:39:08 GMT):
I have updated the CRs up until https://gerrit.hyperledger.org/r/#/c/26670/ to resolve the merge conflicts. @guoger @C0rWin @yacovm You might like to go through your CRs to verify that I haven't inadvertently made some errors during the updates.
guoger (Mon, 08 Oct 2018 09:40:19 GMT):
thanks a lot!
yacovm (Mon, 08 Oct 2018 09:41:20 GMT):
@guoger we are targeting v1.4 as a beta release
yacovm (Mon, 08 Oct 2018 11:57:59 GMT):
@adarshsaraf123 they are still conflicted
yacovm (Mon, 08 Oct 2018 11:58:09 GMT):
I rebased my https://gerrit.hyperledger.org/r/#/c/26038/ on top of master
adarshsaraf123 (Mon, 08 Oct 2018 11:59:11 GMT):
Ok. I will recheck
guoger (Mon, 08 Oct 2018 12:56:09 GMT):
@yacovm could you also post the race condition you found in multi-node support?
yacovm (Mon, 08 Oct 2018 12:57:00 GMT):
haven't looked too deep
yacovm (Mon, 08 Oct 2018 12:57:04 GMT):
https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/5235/console
yacovm (Mon, 08 Oct 2018 12:57:52 GMT):
i just assume it's related since multi-node support is the thing that adds concurrency and my work passed race checks
yacovm (Mon, 08 Oct 2018 12:58:14 GMT):
and the other work (configuration handling) seems to me (in intuiting) as something that wouldn't create race conditions :)
yacovm (Mon, 08 Oct 2018 12:58:17 GMT):
but i might be wrong
guoger (Mon, 08 Oct 2018 12:59:56 GMT):
that's a very fair observation, I'll take a look
guoger (Mon, 08 Oct 2018 13:00:02 GMT):
thx!
adarshsaraf123 (Mon, 08 Oct 2018 19:24:32 GMT):
Screen Shot 2018-10-09 at 12.53.44 AM.png
adarshsaraf123 (Mon, 08 Oct 2018 19:29:02 GMT):
@guoger ^^ Just in case you did not locate this yet, this is from the jenkins build log of https://gerrit.hyperledger.org/r/#/c/25479/.
Also confirmed that it is a deterministic failure since the CRs above this on the stack also fail in the unit tests.
adarshsaraf123 (Mon, 08 Oct 2018 19:35:24 GMT):
And, ironically, for the [decoupling CreateNextBlock and WriteBlock CR](https://gerrit.hyperledger.org/r/#/c/26670/) I worked hard to write test cases that can specifically introduce race conditions!! :sweat_smile:
yacovm (Mon, 08 Oct 2018 19:37:26 GMT):
how do you create a text with a clickable link @adarshsaraf123 ?
adarshsaraf123 (Mon, 08 Oct 2018 19:38:39 GMT):
You use '[text](link)'
adarshsaraf123 (Mon, 08 Oct 2018 19:38:39 GMT):
@yacovm You use '[text](link)'
yacovm (Mon, 08 Oct 2018 19:39:51 GMT):
[thanks](https://en.wiktionary.org/wiki/thanks)
guoger (Tue, 09 Oct 2018 06:01:53 GMT):
> Also confirmed that it is a deterministic failure since the CRs above this on the stack also fail in the unit tests.
@adarshsaraf123 could you confirm which test case is deterministically failing? I couldn't reproduce it locally...
As far as I could tell, there *is* a data race in _test_, and I'm now fixing it.
guoger (Tue, 09 Oct 2018 06:01:53 GMT):
@adarshsaraf123
> Also confirmed that it is a deterministic failure since the CRs above this on the stack also fail in the unit tests.
could you confirm which test case is deterministically failing? I couldn't reproduce it locally...
As far as I could tell, there *is* a data race in _test_, and I'm now fixing it.
adarshsaraf123 (Tue, 09 Oct 2018 06:06:40 GMT):
@guoger did you try with `go test -race` in the `orderer/consensus/etcdraft` package?
adarshsaraf123 (Tue, 09 Oct 2018 06:07:31 GMT):
Screen Shot 2018-10-09 at 11.36.59 AM.png
guoger (Tue, 09 Oct 2018 06:07:40 GMT):
oh, I thought you said a test case is failing. It actually pass, but a race is detected
adarshsaraf123 (Tue, 09 Oct 2018 06:07:41 GMT):
The above is from a local run.
adarshsaraf123 (Tue, 09 Oct 2018 06:08:18 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=97d5vt8BntqFJ9QLh) @guoger Doesn't the race detection fail the test?
guoger (Tue, 09 Oct 2018 06:08:59 GMT):
nvm, I thought you were saying `ginkgo` failed, but not `ginkgo -race`
adarshsaraf123 (Tue, 09 Oct 2018 06:09:12 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=sXpcwvdas8agxjriG) @guoger Ok. Got it.
kostas (Tue, 09 Oct 2018 14:41:53 GMT):
> oh, I thought you said a test case is failing. It actually pass, but a race is detected
Ah, FWIW a test _was_ failing there as well due to a type mismatch, but that was an easy fix.
guoger (Tue, 09 Oct 2018 17:11:10 GMT):
which one?
kostas (Tue, 09 Oct 2018 21:07:41 GMT):
@guoger: https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=GLyARozSonRzjD5nB
kostas (Tue, 09 Oct 2018 21:15:21 GMT):
@guoger: https://gerrit.hyperledger.org/r/c/25479/27/orderer/consensus/etcdraft/chain_test.go#660
kostas (Tue, 09 Oct 2018 21:16:34 GMT):
Setting the Kafka migration work aside (sheet "Weeks 12-14") is there any work in [this spreadsheet](https://docs.google.com/spreadsheets/d/1R02-xjdl4hNw90kN4NZ1kX0f4QoIYn51_1UHlwy-GQs/edit) that we haven't already done or are not currently working on? Please have a look and let me know.
yacovm (Tue, 09 Oct 2018 21:17:27 GMT):
we certainly never tested adding new nodes :(
yacovm (Tue, 09 Oct 2018 21:17:32 GMT):
like - dynamically
yacovm (Tue, 09 Oct 2018 21:17:41 GMT):
unless i'm mistaken
kostas (Tue, 09 Oct 2018 21:18:01 GMT):
I don't think you are.
kostas (Tue, 09 Oct 2018 21:18:29 GMT):
I am going to add this as new sub-task in FAB-11161 before we close it.
kostas (Tue, 09 Oct 2018 21:18:34 GMT):
Thanks for that.
kostas (Tue, 09 Oct 2018 21:26:31 GMT):
We have about 5-6 weeks to nail everything (outside of Kafka migration), so if anything basic is missing, please speak up ;)
kostas (Tue, 09 Oct 2018 21:33:09 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=EwtX2BFMEFj2ah4tx
kostas (Tue, 09 Oct 2018 21:33:33 GMT):
Actually, we have written this down as Scenario 3 in FAB-11590.
kostas (Tue, 09 Oct 2018 21:39:38 GMT):
So we should have this integration test when Artem submits his Type B CR. Good reminder.
grapebaba (Wed, 10 Oct 2018 02:15:14 GMT):
Has joined the channel.
guoger (Wed, 10 Oct 2018 08:26:47 GMT):
am i the only one seeing this [constant ut failure](https://logs.hyperledger.org/production/vex-yul-hyp-jenkins-3/fabric-verify-unit-tests-x86_64/5284/console.log.gz) ?
yacovm (Wed, 10 Oct 2018 08:28:24 GMT):
it's not constant
yacovm (Wed, 10 Oct 2018 08:28:30 GMT):
re-run the UTs
yacovm (Wed, 10 Oct 2018 08:28:34 GMT):
via `Run UnitTest`
guoger (Wed, 10 Oct 2018 08:29:22 GMT):
did it twice.. will try again..
guoger (Wed, 10 Oct 2018 08:33:57 GMT):
are we supposed to use dep v0.5.0 now?
guoger (Wed, 10 Oct 2018 08:34:37 GMT):
running `dep check` on master gives me error
guoger (Wed, 10 Oct 2018 08:34:37 GMT):
running `dep check` on master gives me error (with 0.5.0)
guoger (Wed, 10 Oct 2018 09:54:43 GMT):
nvm, `build` somehow crawls back to my fabric dir and all go files in it are generating toubles...
guoger (Wed, 10 Oct 2018 09:54:43 GMT):
nvm, `build` somehow crawls back to my fabric dir and all go files in it are generating troubles...
adarshsaraf123 (Fri, 12 Oct 2018 03:31:24 GMT):
@yacovm @C0rWin @guoger I am arranging the stack in order. Some CRs are out of date with updates to their parents.
guoger (Fri, 12 Oct 2018 06:24:18 GMT):
@adarshsaraf123 i pushed my wal CRs on top of multi-node support CR. once your decoupling CR is somewhat stable, I'll rebase atop
guoger (Fri, 12 Oct 2018 06:24:42 GMT):
so don't worry about conflict there yet
adarshsaraf123 (Fri, 12 Oct 2018 06:27:07 GMT):
Cool :+1_tone4:
guoger (Fri, 12 Oct 2018 06:40:42 GMT):
so, i'm adding `WALDir` to orderer config and I need suggestion here: as we want to make consensus pluggable, i'm reluctant to add a `Etcdraft` section along with `Kafka` in `orderer.yaml`.
should we:
- add a section there which is opaque to orderer, and
- add a consensus specific parser that understands this section
adarshsaraf123 (Fri, 12 Oct 2018 06:57:08 GMT):
Can there not be `WALDir` that can be configured irrespective of the consensus type? Of course, we don't need it for `solo` and `kafka` but my guess is we will need it for the subsequent BFT as well. I think it will fit well in the `FileLedger` section.
guoger (Fri, 12 Oct 2018 07:01:21 GMT):
in that line of thinking, it probably deserve a separate section for `Consensus Data` or `Replicated Logs` (better names are needed here) other than `FileLedger`.
guoger (Fri, 12 Oct 2018 07:01:21 GMT):
in that line of thinking, it probably deserve a separate section for `Consensus Data` or `Replicated Logs` (better names are needed here) other than `FileLedger`. the same applies to snapshotting
guoger (Fri, 12 Oct 2018 07:29:15 GMT):
cc @kostas
kostas (Fri, 12 Oct 2018 18:12:20 GMT):
> should we:
> - add a section there which is opaque to orderer, and
> - add a consensus specific parser that understands this section
kostas (Fri, 12 Oct 2018 18:12:30 GMT):
@guoger: Expand a bit on how you see this one working?
adarshsaraf123 (Sun, 14 Oct 2018 09:21:35 GMT):
I have rebased the existing stack on master
yacovm (Sun, 14 Oct 2018 10:34:07 GMT):
@guoger I pushed a preliminary version of the block puller and I added some glue code that adds it to the `chain.go` so you can take a look.
https://gerrit.hyperledger.org/r/#/c/26932/11/orderer/consensus/etcdraft/chain.go
Here :arrow_up: is a definition of the interface + adding it to the `etcdraft.Chain` struct
https://gerrit.hyperledger.org/r/#/c/26932/11/orderer/consensus/etcdraft/consenter.go
Here :arrow_up: is how it's added to the `consenter.go`,
I guess we have time until the change sets below will get to the master branch so i'll try to complete the change set this week and also split it to several parts so it'll be easier to review, but - it's pretty straight forward to add this into the `etcdraft.Consenter` from what I see
yacovm (Sun, 14 Oct 2018 10:34:23 GMT):
@kostas @C0rWin @adarshsaraf123 ^
yacovm (Sun, 14 Oct 2018 15:29:06 GMT):
also got a basic (green path UT)[https://gerrit.hyperledger.org/r/#/c/26932/17/orderer/common/cluster/deliver_test.go@171] working
yacovm (Sun, 14 Oct 2018 15:29:23 GMT):
also got a basic [green path UT](https://gerrit.hyperledger.org/r/#/c/26932/17/orderer/common/cluster/deliver_test.go@171) working
yacovm (Sun, 14 Oct 2018 16:40:37 GMT):
https://gerrit.hyperledger.org/r/#/c/26297/ @kostas is there a reason you didn't merge this?
kostas (Sun, 14 Oct 2018 16:43:29 GMT):
@yacovm: Was being a bit conservative to see if Matt had any follow-up comments. Merged now.
kostas (Sun, 14 Oct 2018 16:46:15 GMT):
@yacovm @C0rWin: Have a look at Adarsh's decoupling work ([CR1](https://gerrit.hyperledger.org/r/c/26670/), [CR2](https://gerrit.hyperledger.org/r/c/26880/)) when you get a chance. I'll check the delta between patchsets 18 and 20 tomorrow, but we should try to make a decision towards either merging or revising further tomorrow.
yacovm (Sun, 14 Oct 2018 16:57:48 GMT):
you didn't merge yet @kostas
kostas (Sun, 14 Oct 2018 17:38:18 GMT):
Hmm, indeed I didn't, and that was a slip-up.
kostas (Sun, 14 Oct 2018 17:38:26 GMT):
Here's what I get when I click on `Submit` now:
kostas (Sun, 14 Oct 2018 17:39:11 GMT):
Screenshot 2018-10-14 at 13.37.51.png
kostas (Sun, 14 Oct 2018 17:39:14 GMT):
Any idea what's up?
kostas (Sun, 14 Oct 2018 17:40:13 GMT):
Screenshot 2018-10-14 at 13.39.24.png
yacovm (Sun, 14 Oct 2018 18:18:06 GMT):
try now
yacovm (Sun, 14 Oct 2018 18:18:08 GMT):
@kostas
yacovm (Sun, 14 Oct 2018 18:25:31 GMT):
thanks
baoyangc (Mon, 15 Oct 2018 03:03:41 GMT):
Has joined the channel.
guoger (Mon, 15 Oct 2018 09:07:47 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=g5vAzv3cefGoawyET) @kostas i.e. we could have this in `orderer.yaml`:
```
Consensus:
Type: etcdraft
Config:
WALDir: /path/to/etcdraft/data/wal
```
and define this struct under `TopLevel`:
```
type TopLevel struct {
General General
FileLedger FileLedger
RAMLedger RAMLedger
Kafka Kafka
Debug Debug
Consensus Consensus
}
type Consensus struct {
Type string
Config interface{}
}
type EtcdRaft struct {
WALDir string
}
```
when we decode viper configs into struct, we will decode this section based on `Consensus.Type`. wdyt?
guoger (Mon, 15 Oct 2018 09:40:33 GMT):
guys, I most likely won't be able to attend our scrum today, here's my progress:
snapshotting is currently split into 4 pieces:
- refactor current `chain` to bundle memory storage, wal and snapshot into a `Storage struct` (done)
- enable chain to take snapshots at certain interval (wip)
- enable chain to handle `Snap` messages from raft leader (wip)
- pull blocks using BlockPuller (todo)
decision we need to make:
- how do we configure snapshotting intervals? it can be taken independently across OSN, i.e. Node A takes snapshot every X blocks, Node B takes snapshot every Y blocks. So it could be in `orderer.yaml`. _OR_, it can be the same in a channel.
guoger (Mon, 15 Oct 2018 09:40:33 GMT):
guys, I most likely won't be able to attend our scrum today, my apologies... here's my progress:
snapshotting is currently split into 4 pieces:
- refactor current `chain` to bundle memory storage, wal and snapshot into a `Storage struct` (done)
- enable chain to take snapshots at certain interval (wip)
- enable chain to handle `Snap` messages from raft leader (wip)
- pull blocks using BlockPuller (todo)
decision we need to make:
- how do we configure snapshotting intervals? it can be taken independently across OSN, i.e. Node A takes snapshot every X blocks, Node B takes snapshot every Y blocks. So it could be in `orderer.yaml`. _OR_, it can be the same in a channel.
yacovm (Mon, 15 Oct 2018 09:44:48 GMT):
it shouldn't be per block numbers IMO
yacovm (Mon, 15 Oct 2018 09:44:59 GMT):
it should be per block cumulative size
yacovm (Mon, 15 Oct 2018 09:45:13 GMT):
i.e - once a certain amount of data passed in the chain
yacovm (Mon, 15 Oct 2018 09:45:50 GMT):
also I don't think it should be in orderer.yaml but in the channel config
yacovm (Mon, 15 Oct 2018 09:47:13 GMT):
I see stuff belongs to orderer.yaml only if they are either related to information about the node itself, or related to performance/failure detection configuration such as timeouts, memory buffers, etc.
adarshsaraf123 (Mon, 15 Oct 2018 09:56:55 GMT):
I agree with the snapshotting interval being a channel config.
I also think that the interval should be in terms of the number of blocks and not the cumulative block size. Block size is not a number we can easily think in terms of and also it does not reflect the total number of transactions so I am not really sure if we gain anything by specifying the snapshot interval in terms of the block size.
yacovm (Mon, 15 Oct 2018 09:58:28 GMT):
block size is a very good estimate for number of transactions actually
yacovm (Mon, 15 Oct 2018 09:59:18 GMT):
i think that transactions have a zipf distribution in terms of size
yacovm (Mon, 15 Oct 2018 09:59:35 GMT):
because, the initial size of a transaction is 3KB
yacovm (Mon, 15 Oct 2018 09:59:35 GMT):
because, the initial size of a transaction is ~ 2.5KB
yacovm (Mon, 15 Oct 2018 09:59:53 GMT):
since we have roughly 3 PEMs and each PEM is 0.7KB :)
yacovm (Mon, 15 Oct 2018 09:59:53 GMT):
since we have roughly 3 PEMs and each PEM is 0.7KB :) and most people just want to transfer simple strings and don't put images or blue ray DVDs in the transactions
yacovm (Mon, 15 Oct 2018 10:00:22 GMT):
on the other hand - you might have a noisy channel where a block would be committed once per second even if the timeout is 2 seconds
yacovm (Mon, 15 Oct 2018 10:00:27 GMT):
(say, in kafka)
yacovm (Mon, 15 Oct 2018 10:01:02 GMT):
but - you might have another channel where you have only a single transction in 2 seconds
yacovm (Mon, 15 Oct 2018 10:01:13 GMT):
so you have 2 channels - one with huge blocks, and one with tiny blocks
yacovm (Mon, 15 Oct 2018 10:01:26 GMT):
doesn't make sense to snapshot in the same frequency both channels
yacovm (Mon, 15 Oct 2018 10:01:29 GMT):
my 2 cents
adarshsaraf123 (Mon, 15 Oct 2018 10:07:10 GMT):
I would think that, in this case, the onus is on the orderer admins to correctly configure the snapshot interval taking into account the frequency of block creation in both the channels.
adarshsaraf123 (Mon, 15 Oct 2018 10:09:26 GMT):
Also, as a related question, @yacovm will not the size of the transaction vary based on the number of endorsements required? Since each of the endorsers would have to sign it? Or are you only referring to the general sizes seen?
guoger (Mon, 15 Oct 2018 10:19:02 GMT):
I used block number as an example, it could also be the same way we but blocks, i.e. block number and size. Whichever comes first
guoger (Mon, 15 Oct 2018 10:25:14 GMT):
Also, if we don’t have a raft specific config section, I assume we will have a consensus data dir path in ordered.yaml?
yacovm (Mon, 15 Oct 2018 11:10:55 GMT):
> will not the size of the transaction vary based on the number of endorsements required? Since each of the endorsers would have to sign it?
of course it is related. But that only strengthens my claim that what matters is the size and not the block count, no?
> I would think that, in this case, the onus is on the orderer admins to correctly configure the snapshot interval taking into account the frequency of block creation in both the channels.
There is no onus here... we put a sane default and no one messes with it, or if you want to change it - you just put whatever you want in the `configtx.yaml` and not touch it ever again.
adarshsaraf123 (Mon, 15 Oct 2018 11:58:15 GMT):
>But that only strengthens my claim that what matters is the size and not the block count, no?
Like I was mentioning earlier, IMO, block size is not something that is easy to think in terms of. For example, when I say that a snapshot is taken every 100 blocks it is much easier to get a mental picture of than saying I want a snapshot every 100 MB. Due to the varying block sizes, which follows from the varying transaction sizes, 100 MB could mean 50 blocks in one snapshot period and 100 in another. I think reasoning about a system like this is a little more difficult. Also, if blocks are being cut, let's say 80% of the time, due to a block timeout of 1s then we could even assume that snapshots are taken every 80 seconds. Also, if were to draw references from databases, snapshots are taken every n seconds and the closest we probably have to such a timer is number of blocks.
Of course, I do understand that block size would make more sense in terms of the 'amount of data' that a snapshot contains.
yacovm (Mon, 15 Oct 2018 12:02:07 GMT):
who says that in databases snapshots are done every several seconds?
yacovm (Mon, 15 Oct 2018 12:02:51 GMT):
I don't think it's correct... snapshots for backup in databases from my knowledge are made on demand
yacovm (Mon, 15 Oct 2018 12:03:09 GMT):
when you want to snapshot a database, it stops writing to the data files, and writes only to redo logs
yacovm (Mon, 15 Oct 2018 12:03:19 GMT):
this way the files don't change and you can back them up
yacovm (Mon, 15 Oct 2018 12:03:45 GMT):
then when you finish backing up the file you make the database commit the redo logs into the files back
adarshsaraf123 (Mon, 15 Oct 2018 12:15:43 GMT):
My bad. I meant database checkpoints and not snapshots. And now it makes me wonder if what we are doing for the raft consensus is different from checkpoints. I would think not.
yacovm (Mon, 15 Oct 2018 12:21:58 GMT):
i'm a bit confused. aren't checkpoints for recovery? I thought snapshots in raft are for replication and not recovery.
yacovm (Mon, 15 Oct 2018 12:22:23 GMT):
but in any case... even if we'd want to do a time-based snapshot
yacovm (Mon, 15 Oct 2018 12:22:31 GMT):
you can't use the blocks number for that ;)
kostas (Mon, 15 Oct 2018 14:18:22 GMT):
@guoger: So the benefit of this approach when it comes to config, is that we don't have to keep expanding the file which hosts the `TopLevel` definition with configuration options for _all_ the consensus implementations that Fabric supports.
kostas (Mon, 15 Oct 2018 14:19:10 GMT):
Instead we can only provide the definition (`type EtcdRaft struct...` in your case) for _only_ the implementations that we wish to support.
kostas (Mon, 15 Oct 2018 14:19:21 GMT):
Do I get this right? Any other benefits that I'm overlooking?
kostas (Mon, 15 Oct 2018 14:19:32 GMT):
I'd say I'm onboard BTW - not sure what others think.
kostas (Mon, 15 Oct 2018 14:27:48 GMT):
> how do we configure snapshotting intervals? it can be taken independently across OSN, i.e. Node A takes snapshot every X blocks, Node B takes snapshot every Y blocks. So it could be in `orderer.yaml`. _OR_, it can be the same in a channel.
Nth-ing the motion to make this a channel configuration.
kostas (Mon, 15 Oct 2018 14:27:48 GMT):
> how do we configure snapshotting intervals? it can be taken independently across OSN, i.e. Node A takes snapshot every X blocks, Node B takes snapshot every Y blocks. So it could be in `orderer.yaml`. _OR_, it can be the same in a channel.
Nth-ing the motion to make this a channel configuration item.
kostas (Mon, 15 Oct 2018 14:28:56 GMT):
As for how we define the snapshot interval: I'm torn. Snapshotting per size is the proper thing to do, but there is something to be said about saying "we snapshot every 100 blocks".
kostas (Mon, 15 Oct 2018 14:29:38 GMT):
I don't have a strong preference for this.
kostas (Mon, 15 Oct 2018 14:30:10 GMT):
Let's proceed with whatever is easier for now so that we have something out there.
adarshsaraf123 (Mon, 15 Oct 2018 14:45:41 GMT):
@kostas @yacovm @C0rWin @guoger There is only one lock that I introduce in the decoupling CR and in my opinion it should not create any troubles. However, I can also understand the concern and definitely it is very easy to ignore certain corner cases which will leave us hanging in the dark.
The addition of the blocks enabled the parallel creation and writing of blocks. Now, if that does not seem to be important since it looks like everyone's thinking of having a single goroutine per chain, then we can do away with the second lock. If this seems okay, then I will change the implementation accordingly.
kostas (Mon, 15 Oct 2018 15:09:58 GMT):
@adarshsaraf123: I am good with the current implementation. (I'm always concerned with locks :wink:) If anybody else objects, now's the time to raise that objection.
guoger (Mon, 15 Oct 2018 15:45:31 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=WkhrDGSKK3nQ4qRh3) @kostas I assume you were talking about expanding `orderer.yaml` to contain a `Consensus` section, and have an `EctdRaft struct`?
kostas (Mon, 15 Oct 2018 16:08:44 GMT):
I am talking about `localconfig/config.go`.
kostas (Mon, 15 Oct 2018 16:09:21 GMT):
If we proceed as you suggest, we can omit the definition for `BFT` and we'll still have a working binary.
adarshsaraf123 (Mon, 15 Oct 2018 16:31:09 GMT):
@kostas @yacovm @C0rWin @guoger I also need consensus on whether we have only one goroutine which calls either `CreateNextBlock` or `WriteBlock` or if we would like to have them in separate goroutines. If it will ever only be one goroutine then I can do away with the extra locking in place and make the implementation simpler. I could have erred on going overboard with allowing them to be called parallely.
@guoger I had pointed it out in today's scrum to the others that you had an objection with this. You can comment further here if you would like to.
adarshsaraf123 (Mon, 15 Oct 2018 16:31:09 GMT):
@kostas @yacovm @C0rWin @guoger I also need consensus on whether we have only one goroutine which calls either `CreateNextBlock` or `WriteBlock` or if we would like to have them in separate goroutines. If it will ever only be one goroutine then I can do away with the extra locking in place and make the implementation simpler. I could have erred on going overboard with allowing them to be called parallely.
@guoger I had pointed it out in today's scrum to the others that you had an objection with this. You can comment further here if you would like to.
yacovm (Mon, 15 Oct 2018 16:33:42 GMT):
it's @guoger 's code , he knows best ;)
kostas (Mon, 15 Oct 2018 16:42:56 GMT):
@adarshsaraf123: Do you think it'd be a significant rewrite to switch to one goroutine?
kostas (Mon, 15 Oct 2018 16:43:07 GMT):
If it's not, let's try this out?
yacovm (Mon, 15 Oct 2018 16:48:46 GMT):
so I looked in the code and we only call CreateNextBlock and WriteBlock from a single goroutine
adarshsaraf123 (Mon, 15 Oct 2018 17:31:52 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=7DkkTjknaKw6FjjBR) @kostas No, in fact, it would be a lot simpler if we are not to have multiple goroutines. And upon thinking of it more, I too think it is better to have the simpler implementation. I will submit a patch set with this update. I also think with this we may not have need to have a separate blockwriter for etcd raft and would not do the separation. If after seeing the CR, there is a need felt to create a separate package, i will take care of it then.
adarshsaraf123 (Mon, 15 Oct 2018 17:33:02 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=883bxFnYbbefac5ea) @yacovm True that. I was although thinking of multiple goroutines going ahead. But i admit this is indeed an overkill and it's a lot neater to keep things simple.
adarshsaraf123 (Mon, 15 Oct 2018 18:47:33 GMT):
Have updated the CRs with only a single goroutine. I hope this patch set is a lot simpler and clearer.
akshay.sood (Mon, 15 Oct 2018 19:47:42 GMT):
Has joined the channel.
yacovm (Mon, 15 Oct 2018 19:52:43 GMT):
what about lock count?
yacovm (Mon, 15 Oct 2018 19:52:54 GMT):
still 10 mutexes?
yacovm (Mon, 15 Oct 2018 19:54:04 GMT):
wait i don't understand i thought Kostas wanted a separate CR that adds it.... and doesn't change the existing one, no?
guoger (Tue, 16 Oct 2018 02:33:30 GMT):
haven't looked latest decoupling code yet, but here's what I think based on the conversation in this channel:
we call `create` and `write` in _single go routine_ because:
- in solo and kafka, there's no need to separate them, cuz we are consenting on tx instead of blocks, so `create` and `write` can be done back-to-back.
- in etcdraft, this coupling *is* the reason we call them in single go routine. Once they are decoupled, we can (and most likely should) access them from separate go routines
what I was objecting is that we _shouldn't_ enforce them to be called by different go routines, but @adarshsaraf123 has fixed it after an offline conversation. (in previous implementation, there's a potential deadlock when `create` and `write` are accessed by single go routine)
guoger (Tue, 16 Oct 2018 02:37:52 GMT):
however, I would like us to defer changing the caller to use 2 go routines, cuz i feel there are more to be taken care of than [this](https://gerrit.hyperledger.org/r/c/26880/), i.e. blocking incoming envelopes while a config block is being assembled, consented and written.
guoger (Tue, 16 Oct 2018 02:37:52 GMT):
however, I would like us to defer changing the caller to use 2 go routines, cuz i feel there are more to be taken care of than [this](https://gerrit.hyperledger.org/r/c/26880/), i.e. blocking incoming envelopes while a config block is being assembled, consented and written.
guoger (Tue, 16 Oct 2018 02:39:37 GMT):
I suggest we create a JIRA to log this, and revisit once we are done with a fully functional etcd/raft chain
guoger (Tue, 16 Oct 2018 04:38:39 GMT):
@yacovm in your reply in FAB-12016
> We first pull the chains, and pull the system chain last until it reaches the sequence of the bootstrap config block in the file system, which is used to bootstrap the orderer.
why are we pulling chains _before_ system chain?
adarshsaraf123 (Tue, 16 Oct 2018 05:04:46 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=dyhS3tuyanhoAEQmc) @guoger Oh okay. From our conversations I got the feeling that you were against splitting them out into two goroutines. That said, I am now of the opinion that there is not much that we gain by separating them into two goroutines. There is hardly much being done in `WriteBlock` to justify the additional lock management and complexity in code necessary to allow `CreateNextBlock` to be called in parallel to `WriteBlock`. I think it is better we stick to the single goroutine implementation. We do have the decoupling still though, just without the parallelism though.
adarshsaraf123 (Tue, 16 Oct 2018 05:04:46 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=dyhS3tuyanhoAEQmc) @guoger Oh okay. From our conversations I got the feeling that you were against splitting them out into two goroutines. That said, I am now of the opinion that there is not much that we gain by separating them into two goroutines. There is hardly much being done in `WriteBlock` to justify the additional lock management and complexity in code necessary to allow `CreateNextBlock` to be called in parallel to `WriteBlock`. I think it is better we stick to the single goroutine implementation. We do have the decoupling still, just without the parallelism though.
guoger (Tue, 16 Oct 2018 05:30:05 GMT):
@adarshsaraf123 then may i ask why we are decoupling them at first place?
adarshsaraf123 (Tue, 16 Oct 2018 05:59:06 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=bg2Xg9BShfjyDh49c) @guoger Decoupling was not in the sense of calling `CreateNextBlock` and `WriteBlock` in parallel. With the earlier implementation, we could only create one block and then wait until this block was written before being able to create the next block. However, the present decoupling allows multiple blocks to be created without having to wait for the created blocks to be written. We can create 10 blocks without waiting for any of them to be written, a luxury we could not afford earlier. Perhaps looking at the [follow-up CR](https://gerrit.hyperledger.org/r/#/c/26880/) will help make this clearer?
guoger (Tue, 16 Oct 2018 06:01:33 GMT):
@adarshsaraf123 and if we do not plan to call them in parallel, how do you create new blocks without waiting for the previous one to be written? (actually the one before that, given current impl)
guoger (Tue, 16 Oct 2018 06:02:30 GMT):
wait, I got it..
guoger (Tue, 16 Oct 2018 06:08:31 GMT):
alright, I agree that it doesn't need to be thread-safe _for now_. *but*, you need to handle this case in your follow-up CR:
>blocking incoming envelopes while a config block is being assembled, consented and written
guoger (Tue, 16 Oct 2018 06:09:29 GMT):
cc @adarshsaraf123
adarshsaraf123 (Tue, 16 Oct 2018 06:10:34 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=Zg3ibLEwnnktiETcu) @guoger Actually that's part of a [separate JIRA](https://jira.hyperledger.org/browse/FAB-12226). I will create a sub-task there and work on it.
guoger (Tue, 16 Oct 2018 06:12:09 GMT):
I think we are talking about the [same CR](https://gerrit.hyperledger.org/r/#/c/26880/)
adarshsaraf123 (Tue, 16 Oct 2018 06:13:14 GMT):
It is based on the [Raft JIRA breakdown sheet](https://docs.google.com/spreadsheets/d/1R02-xjdl4hNw90kN4NZ1kX0f4QoIYn51_1UHlwy-GQs/edit#gid=879975188) Week 11.
adarshsaraf123 (Tue, 16 Oct 2018 06:15:06 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=cXdWna7Ww4FrKCfsQ) @guoger For bookkeeping reasons i think it is better that we separate the optimization part from the decoupling. Towards that I was suggesting having a separate JIRA
guoger (Tue, 16 Oct 2018 06:16:23 GMT):
but to utilize decoupling, you *have to* implement this blocking manner, otherwise it's not correct
guoger (Tue, 16 Oct 2018 06:18:24 GMT):
in case we are talking about the same thing again, here's what I meant:
- https://gerrit.hyperledger.org/r/c/26670/ decoupling
- https://gerrit.hyperledger.org/r/c/26880/19 utilize decoupling (blocking is required)
guoger (Tue, 16 Oct 2018 06:19:38 GMT):
and by _follow-up_ CR, I meant the second
adarshsaraf123 (Tue, 16 Oct 2018 07:03:46 GMT):
:+1_tone4:
kostas (Tue, 16 Oct 2018 11:41:34 GMT):
@C0rWin: You have undone my patchset on this one :-( https://gerrit.hyperledger.org/r/c/25479/35..36
kostas (Tue, 16 Oct 2018 11:43:46 GMT):
(I don't want to ripple through your work, so please give it a fix when you get a chance?)
C0rWin (Tue, 16 Oct 2018 11:43:47 GMT):
@kostas not intentionally, just rebased stack of commits and put mine on top
kostas (Tue, 16 Oct 2018 11:43:57 GMT):
Yeah, I know.
kostas (Tue, 16 Oct 2018 11:45:25 GMT):
> why are we pulling chains _before_ system chain?
kostas (Tue, 16 Oct 2018 11:45:29 GMT):
That's a good question.
kostas (Tue, 16 Oct 2018 11:45:36 GMT):
I'm interested in it as well.
yacovm (Tue, 16 Oct 2018 11:46:17 GMT):
if first pull the system channel and only later on pull the chains and then crash
yacovm (Tue, 16 Oct 2018 11:46:23 GMT):
when you restart you are in limbo state
yacovm (Tue, 16 Oct 2018 11:46:30 GMT):
you don't know how to continue
yacovm (Tue, 16 Oct 2018 11:46:30 GMT):
you don't know how to continue - you'll have chains that some are good, and some can never be recovered
yacovm (Tue, 16 Oct 2018 11:46:39 GMT):
therefore you first pull the chains, and then pull the system chain
yacovm (Tue, 16 Oct 2018 11:46:45 GMT):
now, if you crash during this time
yacovm (Tue, 16 Oct 2018 11:46:55 GMT):
when you start, you look at the lastest block sequence of the system chain
yacovm (Tue, 16 Oct 2018 11:47:10 GMT):
if that block sequence is less than the block sequence of the config block of the system chain you were given
yacovm (Tue, 16 Oct 2018 11:47:17 GMT):
it means you crashed during the restart
yacovm (Tue, 16 Oct 2018 11:47:48 GMT):
if it's equal, or higher, then it means you don't need to perform the recovery protocol
C0rWin (Tue, 16 Oct 2018 11:48:35 GMT):
@kostas https://gerrit.hyperledger.org/r/#/c/25479/35..37, done
kostas (Tue, 16 Oct 2018 11:53:14 GMT):
> blocking incoming envelopes while a config block is being assembled, consented and written
kostas (Tue, 16 Oct 2018 11:53:37 GMT):
@guoger @adarshsaraf123: Why does this have to be handled now? Not doubting you're right, just seeking clarification.
adarshsaraf123 (Tue, 16 Oct 2018 11:55:03 GMT):
@C0rWin My [follow-up CR for decoupling](https://gerrit.hyperledger.org/r/#/c/26880/) is in a bit of a flux for now. Me and @guoger had a discussion and we found another corner case that had to be handled. I will rebase yours on top of it after I finalize it. Thanks for the patience :sweat_smile:
adarshsaraf123 (Tue, 16 Oct 2018 11:58:28 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=MZXWj8JbCggAq9W3Y) @kostas May I rephrase that line to 'wait until the proposed config block has been written out to disk before continuing with creation of blocks'. This is necessary since we only do the revalidation of the incoming envelopes for config sequence changes once before the creation of blocks. If we don't hold up the block creation then we will end up having transactions with incorrect config sequences in the subsequent blocks.
kostas (Tue, 16 Oct 2018 11:59:07 GMT):
Makes total sense now. Thanks!
adarshsaraf123 (Tue, 16 Oct 2018 12:59:48 GMT):
Looks like the smoke tests are failing for everybody consistently..
kostas (Tue, 16 Oct 2018 13:00:04 GMT):
Yes, this was on #fabric-release the other day as well.
kostas (Tue, 16 Oct 2018 13:20:28 GMT):
Adarsh and I are going over the logic change in `ResetCreatedBlocks` (formerly `DiscardCreatedBlocks`) here: https://gerrit.hyperledger.org/r/c/26670/26/orderer/common/multichannel/blockwriter.go#114
kostas (Tue, 16 Oct 2018 13:21:13 GMT):
If a proposal fails, retrying is the way to go, correct?
kostas (Tue, 16 Oct 2018 13:22:01 GMT):
Even if you're a network-partitioned leader, and the rest of the cluster has switched to a new leader that you're not aware of, I claim that you should keep retrying to Propose _until_ you get a heartbeat informing you that there's a new leader.
kostas (Tue, 16 Oct 2018 13:22:20 GMT):
And I'm fairly certain that the Raft library itself has the check where proposals from old leaders are discarded.
kostas (Tue, 16 Oct 2018 13:22:36 GMT):
Would you say the above is right? @guoger
kostas (Tue, 16 Oct 2018 13:23:18 GMT):
If it is right, I think the consensus we're reaching with @adarshsaraf123 is that we can go back to the old logic for `DiscardCreatedBlocks` w/o the need for the `onwardsBlockNumber` logic.
yacovm (Tue, 16 Oct 2018 13:23:19 GMT):
> And I'm fairly certain that the Raft library itself has the check where proposals from old leaders are discarded.
this is usual in all consensus algorithms as every propose has an epoch / ballot number attached to it
kostas (Tue, 16 Oct 2018 13:35:56 GMT):
@guoger: Going back to the configuration question, my vote is that we define `Consensus.Config` as an empty interface as you suggested, and we decode it based on `Consensus.Type`.
kostas (Tue, 16 Oct 2018 13:36:11 GMT):
Looking at your WAL CR now.
guoger (Tue, 16 Oct 2018 13:43:01 GMT):
> And I'm fairly certain that the Raft library itself has the check where proposals from old leaders are discarded
As the _partitioned network leader_, `Propose` should not fail, because leader *exists*. This method fails in following conditions:
1. node is stopped (we exit, nothing to be done)
2. this node is removed from cluster (this node does not participate in consensus anymore, mostly likely should be stopped, we don't care)
3. no leader is available
4. proposal forwarding is disabled
guoger (Tue, 16 Oct 2018 13:43:01 GMT):
> And I'm fairly certain that the Raft library itself has the check where proposals from old leaders are discarded
As the _partitioned network leader_, `Propose` does *not* fail, because leader *exists*. This method fails in following conditions:
1. node is stopped (we exit, nothing to be done)
2. this node is removed from cluster (this node does not participate in consensus anymore, mostly likely should be stopped, we don't care)
3. no leader is available
4. proposal forwarding is disabled
guoger (Tue, 16 Oct 2018 13:43:53 GMT):
3&4 might happen at leadership transition phase, where _this node_ was leader. Therefore, we shouldn't retry.
guoger (Tue, 16 Oct 2018 13:43:53 GMT):
3&4 might happen at leadership transition phase, where _this node_ was leader. we shouldn't retry in this case because leader change is already underway (network reconnected and new leader is being elected)
adarshsaraf123 (Tue, 16 Oct 2018 13:49:57 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=3TRW2NHr6S2ouN2jN) @guoger Even in this case it _is_ okay to retry? Of course the retries themselves would likely fail but there is no harm in retrying. Am I right?
guoger (Tue, 16 Oct 2018 13:55:23 GMT):
no, we should not retry because it's definitely going to fail. I'm writing an explanation, which could lengthy
guoger (Tue, 16 Oct 2018 13:55:23 GMT):
no, we should not retry because it's definitely going to fail. I'm writing an explanation, which could be lengthy
guoger (Tue, 16 Oct 2018 13:59:36 GMT):
3 happens in this case:
a. we have 3 node cluster, 1, 2, 3
b. 1 is elected as leader
c. 1 is disconnected (still thinks itself as leader)
d. an envelope is ordered on 1, a new block is created, about to be proposed
e. 1 reconnected, and figured out new leader 2 has been elected, 1 steps down. It's view on leader goes through 1 -> 0 -> 2
f. just when it's 0, `Propose` is called, you get an error
g. if you retry, you fail because either 1) it's still 0, or 2) it's 2, proposal forwarding is disabled and you cannot forward
kostas (Tue, 16 Oct 2018 14:05:22 GMT):
@guoger: We are arguing that retrying will go on until we're notified of a new leader.
kostas (Tue, 16 Oct 2018 14:05:42 GMT):
If the leader is 0 or 2, then we won't retry.
guoger (Tue, 16 Oct 2018 14:05:56 GMT):
if the leader is 1, then `Propose` actual succeeds
kostas (Tue, 16 Oct 2018 14:06:16 GMT):
As it should, right?
kostas (Tue, 16 Oct 2018 14:06:30 GMT):
If the leader is actually 1, then that node's proposals should succeed.
kostas (Tue, 16 Oct 2018 14:06:50 GMT):
If 1 thinks 1 is the leader, but everybody else thinks 2 is the leader, 1 will propose, but everybody else will reject.
kostas (Tue, 16 Oct 2018 14:07:06 GMT):
In both cases we're good, it seems?
guoger (Tue, 16 Oct 2018 14:10:24 GMT):
but you got an error because 1 thinks there's *no* leader.
kostas (Tue, 16 Oct 2018 14:11:23 GMT):
Then I stop proposing.
kostas (Tue, 16 Oct 2018 14:12:47 GMT):
Ah, I think I see the issue.
kostas (Tue, 16 Oct 2018 14:12:51 GMT):
> We are arguing that retrying will go on until we're notified of a new leader.
kostas (Tue, 16 Oct 2018 14:12:55 GMT):
Let me rephrase:
kostas (Tue, 16 Oct 2018 14:13:05 GMT):
As long as we are the leader, we keep proposing.
kostas (Tue, 16 Oct 2018 14:13:10 GMT):
Does this work better?
kostas (Tue, 16 Oct 2018 14:13:20 GMT):
Are we still missing the substance?
guoger (Tue, 16 Oct 2018 14:13:37 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=cn4mpRMvzs7fzjhe5) @kostas and you keep succeeding... no retry needed
kostas (Tue, 16 Oct 2018 14:13:51 GMT):
I am partitioned from the network.
kostas (Tue, 16 Oct 2018 14:14:00 GMT):
How does the proposal succeed?
guoger (Tue, 16 Oct 2018 14:14:15 GMT):
`Propose` succeeds, but never got committed
guoger (Tue, 16 Oct 2018 14:14:25 GMT):
`Propose` method returns nil error
guoger (Tue, 16 Oct 2018 14:14:52 GMT):
just never comes out in `CommittedEntries` in `Ready()`
yacovm (Tue, 16 Oct 2018 14:15:02 GMT):
how is that possible?
kostas (Tue, 16 Oct 2018 14:15:09 GMT):
Point me to that line in the code?
yacovm (Tue, 16 Oct 2018 14:15:16 GMT):
if propose succeeded then a quorum agreed on the block
yacovm (Tue, 16 Oct 2018 14:15:39 GMT):
i assume by succeed you mean it goes through all the stages of consensus
yacovm (Tue, 16 Oct 2018 14:15:53 GMT):
then it has to come out of the committed entries
guoger (Tue, 16 Oct 2018 14:15:54 GMT):
no, by succeed, I mean `Propose` returns nil
guoger (Tue, 16 Oct 2018 14:16:10 GMT):
but it does *not* guarantee it's going to be committed
guoger (Tue, 16 Oct 2018 14:16:13 GMT):
it's not blocking
yacovm (Tue, 16 Oct 2018 14:16:25 GMT):
ah ok
yacovm (Tue, 16 Oct 2018 14:16:38 GMT):
then you mean you sent a propose via Step()
yacovm (Tue, 16 Oct 2018 14:16:39 GMT):
that's all
guoger (Tue, 16 Oct 2018 14:16:47 GMT):
exactly
kostas (Tue, 16 Oct 2018 14:16:57 GMT):
Shouldn't Step fail if I'm partitioned?
guoger (Tue, 16 Oct 2018 14:17:03 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=qD8wXj3e96SDBTSXz) @kostas do you mean code in fabric or etcd/raft?
yacovm (Tue, 16 Oct 2018 14:17:15 GMT):
it should because it's an RPC and not a stream
kostas (Tue, 16 Oct 2018 14:17:17 GMT):
@guoger: Fabric.
yacovm (Tue, 16 Oct 2018 14:17:23 GMT):
had it been a stream it wouldn't :)
yacovm (Tue, 16 Oct 2018 14:17:41 GMT):
because stream Send() is async (I read too much of gRPC code in these months...)
yacovm (Tue, 16 Oct 2018 14:17:48 GMT):
it's async up to 64KB
yacovm (Tue, 16 Oct 2018 14:18:43 GMT):
this is the max quota of bytes to put in the http/2 buffer
guoger (Tue, 16 Oct 2018 14:19:50 GMT):
@kostas https://github.com/hyperledger/fabric/blob/master/orderer/consensus/etcdraft/chain.go#L367
guoger (Tue, 16 Oct 2018 14:20:19 GMT):
but fabric code doesn't help much actually.. how do you detect yourself as partitioned from rest of network?
kostas (Tue, 16 Oct 2018 14:21:06 GMT):
Sigh, I'm confused. Zoom?
guoger (Tue, 16 Oct 2018 14:21:10 GMT):
sure
guoger (Tue, 16 Oct 2018 14:21:20 GMT):
grabbing earphone
kostas (Tue, 16 Oct 2018 14:21:33 GMT):
https://zoom.us/j/7432937602
kostas (Tue, 16 Oct 2018 14:35:42 GMT):
OK, discussed with Jay. He came up with what I think is a very nice idea.
kostas (Tue, 16 Oct 2018 14:39:39 GMT):
The way we're currently doing things is: maintain in the blockwriter a queue of blocks that are not yet chained to each other.
kostas (Tue, 16 Oct 2018 14:39:39 GMT):
The way we're currently doing things is: maintain in the blockwriter a queue of blocks ~that are not yet chained to each other~. (This is incorrect, see below.)
kostas (Tue, 16 Oct 2018 14:40:46 GMT):
Why not maintain a candidate _chain_ of blocks at the leader (under the `etcdraft` package)?
kostas (Tue, 16 Oct 2018 14:42:03 GMT):
Whenever a new block is created at the leader, we stored in our queue complete with the hash that links it to the previous block, right there on the spot. (Hence my reference to a chain of blocks.)
kostas (Tue, 16 Oct 2018 14:43:58 GMT):
This means that `CreateNextBlock` now needs two arguments: (blockToBeCreated, prevBlock)
yacovm (Tue, 16 Oct 2018 14:44:04 GMT):
i thought that was where we wanted to go anyway?
kostas (Tue, 16 Oct 2018 14:46:42 GMT):
Gimme a sec to clarify this a bit further. I was in a rush to capture this right after the call, but I need to go over a few details first. Will look at the code and BRB.
guoger (Tue, 16 Oct 2018 15:04:01 GMT):
i would say `CreateNextBlock` is most likely to be an utility function, which does not need knowledge of `BlockWriter`. Basically, node can propose however it wants, and we just write it into ledger once it's consented.
adarshsaraf123 (Tue, 16 Oct 2018 15:21:49 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=QBKs8ozRCvEYNcRyw) @kostas I know you said you want to clarify this further but the queue maintained in the blockwriter is in fact a chain.
adarshsaraf123 (Tue, 16 Oct 2018 15:24:11 GMT):
And when I mention divergent blocks in the decoupling CR I mean a block that is diverging from this chain.
kostas (Tue, 16 Oct 2018 15:56:45 GMT):
> I know you said you want to clarify this further but the queue maintained in the blockwriter is in fact a chain.
kostas (Tue, 16 Oct 2018 15:56:54 GMT):
@adarshsaraf123: Yes, you are right and I'm wrong.
kostas (Tue, 16 Oct 2018 15:57:16 GMT):
When I was writing this, I realized I may have jumped the gun a bit, which is why I stopped and wanted to check the code again.
kostas (Tue, 16 Oct 2018 15:57:42 GMT):
Still, I think the proposal is worth considering. Reframing it the way @guoger put it:
kostas (Tue, 16 Oct 2018 15:58:10 GMT):
Should we have all of this logic in the blockwriter?
kostas (Tue, 16 Oct 2018 16:00:44 GMT):
I suggest the following so as to resolve the dilemma, and be done with this.
kostas (Tue, 16 Oct 2018 16:01:23 GMT):
Is there any time before 9:30pm EDT today that would work for us to have a Zoom call and settle on a final direction for this?
kostas (Tue, 16 Oct 2018 16:01:23 GMT):
Is there any time ~before~ by 9:30pm EDT today that would work for us to have a Zoom call and settle on a final direction for this?
kostas (Tue, 16 Oct 2018 16:01:34 GMT):
If so, let me know. @guoger @adarshsaraf123
kostas (Tue, 16 Oct 2018 16:01:55 GMT):
Otherwise, I can get on a call at 6am EDT tomorrow morning if needed.
adarshsaraf123 (Tue, 16 Oct 2018 16:03:42 GMT):
I am okay with a time that is suitable to @guoger He is the one further away on the timezone ;)
kostas (Tue, 16 Oct 2018 16:04:46 GMT):
I will keep checking throughout the day to see if he's available tonight. Otherwise, tomorrow morning.
kostas (Tue, 16 Oct 2018 16:04:57 GMT):
Thanks Adarsh, and sorry for the back and forth on this one.
adarshsaraf123 (Tue, 16 Oct 2018 16:05:53 GMT):
There's nothing to be sorry about. This has also been fun in a way ;)
adarshsaraf123 (Tue, 16 Oct 2018 16:14:52 GMT):
@kostas Just realized that anytime beyond 3pm edt before 9:30 pm edt would be late night and early morning for me. So something before that would be good.
guoger (Tue, 16 Oct 2018 16:25:03 GMT):
my availability: 8:30 am EDT onwards @adarshsaraf123 @kostas
guoger (Tue, 16 Oct 2018 16:27:19 GMT):
> I know you said you want to clarify this further but the queue maintained in the blockwriter is in fact a chain.
that's right. I was just suggesting that instead of maintaining queue in blockwriter, we could do that in chain, to simplify the management and avoid complicated api. Anyway, let's discuss over the call
kostas (Tue, 16 Oct 2018 16:31:57 GMT):
We'll do 8:30am tomorrow if that works for Adarsh.
adarshsaraf123 (Tue, 16 Oct 2018 16:33:11 GMT):
Works for me :+1_tone3:
kostas (Tue, 16 Oct 2018 20:33:41 GMT):
https://gerrit.hyperledger.org/r/c/25479/
kostas (Tue, 16 Oct 2018 20:33:45 GMT):
https://gerrit.hyperledger.org/r/c/26547/
kostas (Tue, 16 Oct 2018 20:34:35 GMT):
Yacov/Artem: Both of these look mergeable to me. If you agree, let's get them in.
yacovm (Tue, 16 Oct 2018 20:36:04 GMT):
my name is on the signed off by
yacovm (Tue, 16 Oct 2018 20:36:15 GMT):
i generally agree
yacovm (Tue, 16 Oct 2018 20:36:17 GMT):
skimmed it today
kostas (Tue, 16 Oct 2018 20:37:08 GMT):
My name is on the signed-off-by as well, this is not a blocker.
kostas (Tue, 16 Oct 2018 20:37:13 GMT):
If you agree, +2 and merge.
yacovm (Tue, 16 Oct 2018 20:38:30 GMT):
i will merge after a thorough review, Kostas
yacovm (Tue, 16 Oct 2018 20:38:36 GMT):
I'm currently battling my UTs
kostas (Tue, 16 Oct 2018 20:39:06 GMT):
I'm not rushing a review.
C0rWin (Tue, 16 Oct 2018 20:39:57 GMT):
I've actually used both of these for integration testing of type B tx
C0rWin (Tue, 16 Oct 2018 20:40:36 GMT):
looks fine, so merged, if @yacovm will have concerns we might address them in following CRs
kostas (Tue, 16 Oct 2018 20:40:52 GMT):
Agreed, thank you both for looking into it!
C0rWin (Tue, 16 Oct 2018 20:42:01 GMT):
btw, turns out integration framework needs to be updated to support type B updates as well
C0rWin (Tue, 16 Oct 2018 20:42:05 GMT):
https://gerrit.hyperledger.org/r/#/c/27040/
C0rWin (Tue, 16 Oct 2018 20:42:34 GMT):
mod policy for type is orderer admin, so need to be able to sign update with orderer org admin user
kostas (Tue, 16 Oct 2018 20:43:07 GMT):
A-ha. Cool, thanks for pushing this!
C0rWin (Tue, 16 Oct 2018 20:44:02 GMT):
looks like CI is complete total lose :(
C0rWin (Tue, 16 Oct 2018 20:45:01 GMT):
trying to solve protos divergence, I mean not solve, but rather push... fails all the way on not related to updated things, seriously considering to put +1 for verification myself
kostas (Tue, 16 Oct 2018 20:48:15 GMT):
I was following the discussion in #fabric-scrum earlier today w/ the smoke tests failing. Not good
kostas (Tue, 16 Oct 2018 20:48:15 GMT):
I was following the discussion in #fabric-scrum earlier today w/ the smoke tests failing. Not good.
guoger (Wed, 17 Oct 2018 08:03:38 GMT):
question, as `commitBlock` is async, wouldn't we encounter problem in following case:
- about to `WriteConfigBlock` to create a new chain `foo`
- `newChain` is [invoked](https://github.com/hyperledger/fabric/blob/release-1.3/orderer/common/multichannel/blockwriter.go#L118) before `commitBlock`.
- local directory is created as part of chain resources, *but* config block is *not* yet appended to ledger
- orderer crash
- upon reboot, it finds chain `foo` because it exists on disk, however the config block does not exist on system chain.
guoger (Wed, 17 Oct 2018 08:04:42 GMT):
basically, shouldn't _appending block to ledger_ and _actual creation of chain_ be atomic?
adarshsaraf123 (Wed, 17 Oct 2018 09:56:47 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=9PmjC3L9cqqf4LD8r) @guoger I will second you on that.
Just to add to the discussion what reasoning I could come up with upon finding out `commitBlock` was asynchronous:
Any error in `commitBlock` causes a panic. Perhaps it was worth the performance gains to ignore such corner cases while allowing the normal path to be _significantly_ faster. Maybe it is also reasonable to expect that such anomalies will be taken care of while restarting from a crash (although I am not sure if we do have a checklist to take care of while recovering from an orderer crash).
Also, it _may_ again be reasonable to expect that the append of the genesis block for the new chain will also fail given that the append of the block on the system chain failed.
guoger (Wed, 17 Oct 2018 10:05:53 GMT):
cc @jyellick
guoger (Wed, 17 Oct 2018 12:28:56 GMT):
@kostas @adarshsaraf123 are we going to have the discussion?
kostas (Wed, 17 Oct 2018 12:29:10 GMT):
Yes, meeting in 1' right?
adarshsaraf123 (Wed, 17 Oct 2018 12:29:21 GMT):
👍🏽
yacovm (Wed, 17 Oct 2018 12:29:52 GMT):
what meeting @kostas ?
kostas (Wed, 17 Oct 2018 12:30:34 GMT):
Ah, we posted here yesterday that we wanted to talk about the blockwriter so as to settle the direction we're going.
kostas (Wed, 17 Oct 2018 12:30:39 GMT):
You're welcome to join.
yacovm (Wed, 17 Oct 2018 12:30:40 GMT):
ok
kostas (Wed, 17 Oct 2018 12:30:42 GMT):
https://zoom.us/j/7432937602
kostas (Wed, 17 Oct 2018 13:11:42 GMT):
We reached the 40m limit, restarting
guoger (Wed, 17 Oct 2018 13:48:21 GMT):
@adarshsaraf123 so, my objection against `BlockWriter` is not strong enough to have you rewrite it, but I just wanna suggest that, we probably should avoid making assumptions, and keep it simple (dumb), until we see it being useful. (`BlockWriter` API itself is an example, isn't it :P)
guoger (Wed, 17 Oct 2018 13:52:07 GMT):
but if you see it being over-complicated to shift logic to chain, let's keep your current approach
adarshsaraf123 (Wed, 17 Oct 2018 13:56:13 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=BEiNqAWHpLqGWrTnf) @guoger Fair enough.
I believe there shouldn't be much of a rewrite now that the `GetBlock` call is in. Also, the statement that _it is up to the consensus implementation to decide how it wants to create the blocks_ makes a lot of sense and now I am convinced that this logic sits well within `etcdraft`. The main argument for me against moving it out of `blockwriter` was that we may need it for subsequent consensus implementations but that _is_ an assumption and like you said we _should_ avoid making assumptions :)
adarshsaraf123 (Wed, 17 Oct 2018 13:56:13 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=BEiNqAWHpLqGWrTnf) @guoger Fair enough.
I believe there shouldn't be much of a rewrite now that the `GetBlock` call will be in. Also, the statement that _it is up to the consensus implementation to decide how it wants to create the blocks_ makes a lot of sense and now I am convinced that this logic sits well within `etcdraft`. The main argument for me against moving it out of `blockwriter` was that we may need it for subsequent consensus implementations but that _is_ an assumption and like you said we _should_ avoid making assumptions :)
adarshsaraf123 (Wed, 17 Oct 2018 13:56:13 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=BEiNqAWHpLqGWrTnf) @guoger Fair enough.
I believe there shouldn't be much of a rewrite now that the `GetBlock` call will be in. Also, the statement that _it is up to the consensus implementation to decide how it wants to create the blocks_ makes a lot of sense and now I am pretty convinced that this logic sits well within `etcdraft`. The main argument for me against moving it out of `blockwriter` was that we may need it for subsequent consensus implementations but that _is_ an assumption and like you said we _should_ avoid making assumptions :)
jyellick (Wed, 17 Oct 2018 15:45:50 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=9PmjC3L9cqqf4LD8r
jyellick (Wed, 17 Oct 2018 15:47:31 GMT):
@guoger Absolutely, this is a bug/risk and it should be addressed. There probably should be a comment in the code to that affect. At the time it seemed like a rather unlikely corner case (and recovery in the Kafka case is as simple as deleting the ledger and rebootstrapping). At the very least, we should probably put a check during chain creation time that the chain is already of length zero.
jyellick (Wed, 17 Oct 2018 15:47:56 GMT):
With that check... it's not obvious to me how replaying the orderer system channel would be problematic.
jyellick (Wed, 17 Oct 2018 15:48:19 GMT):
(Of course, making the operation entirely atomic would be superior, but I think we could get away without)
guoger (Thu, 18 Oct 2018 02:37:58 GMT):
> put a check during chain creation time that the chain is already of length zero.
@jyellick mind elaborating a bit?
guoger (Thu, 18 Oct 2018 02:37:58 GMT):
> put a check during chain creation time that the chain is already of length zero.
mind elaborating a bit? @jyellick
guoger (Thu, 18 Oct 2018 03:02:34 GMT):
btw, filed FAB-12521 for this
kostas (Thu, 18 Oct 2018 04:48:06 GMT):
Heads-up: I _may_ miss tomorrow’s scrum. I suggest we use Slack Audio so this can go on w/o me if need be.
C0rWin (Thu, 18 Oct 2018 09:52:01 GMT):
guys, I remember we have discussed at some point, how we suppose to address Raft reconfiguration. I.e. adding or removing OSNs. As far as I can recall, @yacovm suggested to simply stop old instance and create a new one with updated list of peers.
@kostas @guoger @adarshsaraf123 is there other suggestions or proposals?
C0rWin (Thu, 18 Oct 2018 09:52:01 GMT):
guys, I remember we have discussed at some point, how we suppose to address Raft reconfiguration. I.e. adding or removing OSNs. As far as I can recall, @yacovm suggested to simply stop old instance and create a new one with updated list of peers.
@kostas @guoger @adarshsaraf123 is there other suggestions or proposals?
C0rWin (Thu, 18 Oct 2018 09:52:01 GMT):
guys, I remember we have discussed at some point, how we suppose to address Raft reconfiguration. I.e. adding or removing OSNs. As far as I can recall, @yacovm suggested to simply stop old instance and create a new one with updated list of peers.
@kostas @guoger @adarshsaraf123 is there other suggestions or proposals?
kh.nguyen (Fri, 19 Oct 2018 00:21:56 GMT):
Has joined the channel.
yacovm (Fri, 19 Oct 2018 00:48:05 GMT):
@guoger - I finished the CR that adds support for creation of block pullers, and since I see that the `chain.go` and `consenter.go` is a bit ballooning, I added a new file - `util.go` in the `etcdraft` folder.
you may use the `newBlockPuller` method in https://gerrit.hyperledger.org/r/#/c/27055/13/orderer/consensus/etcdraft/util.go@75 to create the block puller for your chain.
The best place to put it, IMO, is in the `HandleChain` method.
You'll need to re-use the `*cluster.PredicateDialer` instance from the communication layer (don't worry - it knows to clone itself so it's safe), and also pass the `localconfig.Cluster` from the top level.
yacovm (Fri, 19 Oct 2018 00:51:36 GMT):
and now I need to ::sleeping:
yacovm (Fri, 19 Oct 2018 00:56:41 GMT):
edit: `03:55:50 orderer/consensus/kafka/chain_test.go:3644: unreachable code` I'll fix it tomorrow :/
Jgnuid (Fri, 19 Oct 2018 01:43:20 GMT):
Has left the channel.
guoger (Fri, 19 Oct 2018 09:24:19 GMT):
I'm seeing this running `make basic-checks` on master:
```
Checking with gofmt
The following files contain gofmt errors
bccsp/pkcs11/impl_test.go
bccsp/sw/impl_test.go
bccsp/utils/x509_test.go
common/cauthdsl/policyparser.go
common/tools/idemixgen/idemixca/idemixca.go
core/endorser/endorser.go
core/endorser/endorser_test.go
core/handlers/validation/builtin/v12/validation_logic_test.go
core/handlers/validation/builtin/v13/validation_logic_test.go
core/ledger/kvledger/txmgmt/rwsetutil/rwset_builder_test.go
core/peer/deliverevents.go
discovery/support/acl/support.go
discovery/support/acl/support_test.go
gossip/gossip/channel/channel.go
gossip/gossip/channel/channel_test.go
gossip/privdata/coordinator.go
integration/nwo/network.go
msp/configbuilder.go
The gofmt command 'gofmt -l -s -w' must be run for these files
make: *** [linter] Error 1
```
can anyone confirm?
guoger (Fri, 19 Oct 2018 09:24:43 GMT):
(but running gofmt doesn't really alter files...)
yacovm (Fri, 19 Oct 2018 09:32:01 GMT):
everything is fine in CI otherwise it'd fail
yacovm (Fri, 19 Oct 2018 09:32:31 GMT):
https://gerrit.hyperledger.org/r/#/c/27102/1/sampleconfig/orderer.yaml
yacovm (Fri, 19 Oct 2018 09:32:36 GMT):
@guoger what is this?
guoger (Fri, 19 Oct 2018 09:33:30 GMT):
CI does fail https://logs.hyperledger.org/production/vex-yul-hyp-jenkins-3/fabric-verify-build-checks-x86_64/6279/console.log.gz
guoger (Fri, 19 Oct 2018 09:34:23 GMT):
that CR is to pass consensus specific configurations to consenter
guoger (Fri, 19 Oct 2018 09:34:37 GMT):
i.e. WAL dir and Snapshot dir
yacovm (Fri, 19 Oct 2018 09:34:59 GMT):
but "hello world" ?
guoger (Fri, 19 Oct 2018 09:35:36 GMT):
if we say `WALDir`, then it would be misleading that it's raft specific
yacovm (Fri, 19 Oct 2018 09:35:52 GMT):
right but why not just add a raft section instead
yacovm (Fri, 19 Oct 2018 09:36:00 GMT):
what does "hello world" mean? :)
guoger (Fri, 19 Oct 2018 09:37:01 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=4HqdJbAzyz2dr76XZ) @yacovm because we don't want a raft section?
guoger (Fri, 19 Oct 2018 09:37:19 GMT):
it doesn't mean anything, because orderer doesn't care
yacovm (Fri, 19 Oct 2018 09:37:20 GMT):
and TBH I think at this stage where we are - golang plugins isn't viable and since we have kafka section we should just add an etcdraft section and call it a day
yacovm (Fri, 19 Oct 2018 09:37:42 GMT):
like - if we have kafka section then it's not generic anyhow
yacovm (Fri, 19 Oct 2018 09:38:03 GMT):
shouldn't we defer pluggability to 2.0 ?
yacovm (Fri, 19 Oct 2018 09:38:09 GMT):
we have so much left to be done
yacovm (Fri, 19 Oct 2018 09:38:30 GMT):
and, while you're here - I finished my work for the deliver client support
yacovm (Fri, 19 Oct 2018 09:38:35 GMT):
Clipboard - October 19, 2018 12:38 PM
yacovm (Fri, 19 Oct 2018 09:38:46 GMT):
may you please rebase on top of my change sets to not conflict me? :)
yacovm (Fri, 19 Oct 2018 09:39:16 GMT):
and you need to pick up the deliver stuff anyway for the snapshotting i believe
yacovm (Fri, 19 Oct 2018 09:39:36 GMT):
I removed the interfaces from the etcdraft chain/consenter so you can easily rebase without any conflicts
yacovm (Fri, 19 Oct 2018 09:39:50 GMT):
as I added code in a new file `util.go`
guoger (Fri, 19 Oct 2018 09:46:26 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=4HqdJbAzyz2dr76XZ) @yacovm I thought we had discussion here earlier? even though we don't care about plugin right now, I'm inclined to add opaque one instead of `EtcdRaft`, we don't need to expand config.
guoger (Fri, 19 Oct 2018 09:46:46 GMT):
and `kafka` should be this way as well, we just don't refactor that for now
yacovm (Fri, 19 Oct 2018 09:52:49 GMT):
I'm not sure it has any value at this point in time given we all know how hard it is to actually add a new consensus to fabric.
if you strongly believe this is optimal, please do so, but please rebase on top of my CR stack :smiling_imp:
yacovm (Fri, 19 Oct 2018 09:52:49 GMT):
I'm not sure it has any value at this point in time given we all know how hard it is to actually add a new consensus to fabric.
The code base is just not ready for pluggability IMO.
if you strongly believe this is optimal, please do so, but please rebase on top of my CR stack :smiling_imp:
guoger (Fri, 19 Oct 2018 10:04:11 GMT):
@C0rWin I'm rebasing on top of yours, do you want me to rebase yours on top of @yacovm 's CR?
yacovm (Fri, 19 Oct 2018 10:06:30 GMT):
hold on, maybe i should move his CR on top of my stack? :thinking:
yacovm (Fri, 19 Oct 2018 10:06:42 GMT):
then you'll end up getting all the goodness
guoger (Fri, 19 Oct 2018 10:08:07 GMT):
`wal` has to be on top of his. `snap` has to be on top of yours. I'm rebasing `wal` right now
guoger (Fri, 19 Oct 2018 10:08:25 GMT):
https://gerrit.hyperledger.org/r/c/27018/
yacovm (Fri, 19 Oct 2018 10:08:33 GMT):
wait hold on
yacovm (Fri, 19 Oct 2018 10:08:39 GMT):
I'll rebase his on top of my stack
yacovm (Fri, 19 Oct 2018 10:09:02 GMT):
then if we end up CRing his before mine, we'll rebase it on top of master, merge it, and then I'll cascade-rebase from gerrit
guoger (Fri, 19 Oct 2018 10:10:08 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=LzTMXWxrmJvFY6GmT) @yacovm that's what I was trying to do
yacovm (Fri, 19 Oct 2018 10:10:34 GMT):
oh then be my guest
yacovm (Fri, 19 Oct 2018 10:10:46 GMT):
feel free to do that
yacovm (Fri, 19 Oct 2018 10:35:01 GMT):
nevermind I did it myself @guoger
yacovm (Fri, 19 Oct 2018 10:35:25 GMT):
Clipboard - October 19, 2018 1:35 PM
guoger (Fri, 19 Oct 2018 10:35:29 GMT):
thx!
yacovm (Fri, 19 Oct 2018 10:36:19 GMT):
@guoger @adarshsaraf123 @C0rWin please everyone keep in mind rebasing on top of this chain so we can slide this to master as smoothly as possible
yacovm (Fri, 19 Oct 2018 10:36:30 GMT):
@kostas ^
kostas (Fri, 19 Oct 2018 12:21:58 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=HDEGQyqubDMeANrAg
kostas (Fri, 19 Oct 2018 12:26:08 GMT):
It is hard, and we botched things with the way we hard-coded Kafka. So long as adding new things in an opaque way is not slowing us down, we should do it like so.
kostas (Fri, 19 Oct 2018 12:26:08 GMT):
It is hard, and we botched things with the way we hard-coded Kafka. So long as adding new things in an opaque way is not slowing us down however, we should do it like so.
kostas (Fri, 19 Oct 2018 12:27:08 GMT):
Put differently: if we can do the right thing w/o it being a significant detour, let's do the right thing. That opaque consensus section strikes me as the right thing.
guoger (Fri, 19 Oct 2018 12:28:39 GMT):
@adarshsaraf123 could you insert your optimistic block creation somewhere in the CR stack? it's rebased
adarshsaraf123 (Fri, 19 Oct 2018 12:30:22 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=8FeWdnYnFCS6fnFpx) @guoger I have not completed working on it yet. I can rebase it wherever in the stack you would like me to.
kostas (Fri, 19 Oct 2018 12:42:17 GMT):
@guoger: Don't let that derail you from your current line of work, a heads up that we had an integration test failure related to the "stale leader should not be able to propose block because of lagged term" spec.
kostas (Fri, 19 Oct 2018 12:42:17 GMT):
@guoger: Don't let that derail you from your current line of work, but a heads up that we had an integration test failure related to the "stale leader should not be able to propose block because of lagged term" spec.
kostas (Fri, 19 Oct 2018 12:42:36 GMT):
(It showed up in the WAL CR, but was not introduced by it.)
guoger (Fri, 19 Oct 2018 12:47:19 GMT):
i remember it's in UT, not integration. There was one failed *integration* test, but fixed in latest CR (basically I didn't pass WALDir to it)
guoger (Fri, 19 Oct 2018 12:47:52 GMT):
and the UT you pointed out, yes, I was aware of it... and trying to reproduce..
kostas (Fri, 19 Oct 2018 12:53:22 GMT):
Right, right - I stand corrected; UT.
yacovm (Sun, 21 Oct 2018 20:47:39 GMT):
why can't https://gerrit.hyperledger.org/r/#/c/26180/ be rebased on top of its parent ? :thinking:
yacovm (Sun, 21 Oct 2018 20:48:15 GMT):
@guoger
yacovm (Sun, 21 Oct 2018 20:48:20 GMT):
any idea?
yacovm (Sun, 21 Oct 2018 20:49:07 GMT):
btw heads up - i rebased most of the CRs in the chain until Jay's change set above.
I had a brittle test so I had to change some lines in a file of mine which I added
guoger (Mon, 22 Oct 2018 03:04:31 GMT):
@yacovm i was, until Artem changed [his](https://gerrit.hyperledger.org/r/c/27108/4), and I'm rebasing it
guoger (Mon, 22 Oct 2018 10:04:12 GMT):
@yacovm I agree with you that we should take snapshot *every n bytes*, instead of every n blocks. However, do you mind if we defer this, cuz it adds a bit complexity:
every time we reboot an OSN, we need to load blocks since last snapshotted and calculate total bytes, so we have a correct initial state. (it should also be straightforward with newly added `Block` api)
yacovm (Mon, 22 Oct 2018 10:11:21 GMT):
I don't understand... if you boot the OSN you just load whatever you have no?
yacovm (Mon, 22 Oct 2018 10:11:28 GMT):
you don't snapshot when you boot
yacovm (Mon, 22 Oct 2018 10:11:32 GMT):
so where is the added complexity at boot?
yacovm (Mon, 22 Oct 2018 10:21:59 GMT):
@guoger
guoger (Mon, 22 Oct 2018 10:22:40 GMT):
we need to know how many bytes of blocks we have accumulated since last snapshot
guoger (Mon, 22 Oct 2018 10:23:43 GMT):
i.e. if we snapshotted at block 5, and then produced 6 & 7, then reboot. we need to have total size of 6 & 7 as initial state. Does it make sense?
guoger (Mon, 22 Oct 2018 10:24:34 GMT):
if we go with just snapshotting every n blocks, then I'll just load block number from last snapshot
guoger (Mon, 22 Oct 2018 10:25:01 GMT):
@yacovm
yacovm (Mon, 22 Oct 2018 11:07:44 GMT):
oh i see you need to persist the last block index @guoger right?
guoger (Mon, 22 Oct 2018 11:19:12 GMT):
by "last block index", do you mean "lasted snapshotted block number"? we don't need that in block metadata. we could get it from last snapshot
yacovm (Mon, 22 Oct 2018 11:21:34 GMT):
so what is the problem then? you can retrieve any block using the support now
yacovm (Mon, 22 Oct 2018 11:21:46 GMT):
you can just retrieve the blocks when you start up and count their lengths
guoger (Mon, 22 Oct 2018 12:28:08 GMT):
oh, there's no problem per se. just a bit more complicated (code and tests) than "snapshot per n blocks", and I think we could defer it to a later improvement. (also, does it make sense to provide both, just like how we cut blocks?)
yacovm (Mon, 22 Oct 2018 12:41:42 GMT):
@kostas ? wdyt?
yacovm (Mon, 22 Oct 2018 13:08:20 GMT):
https://jira.hyperledger.org/browse/FAB-12016
kostas (Mon, 22 Oct 2018 13:38:57 GMT):
> oh, there's no problem per se. just a bit more complicated (code and tests) than "snapshot per n blocks", and I think we could defer it to a later improvement. (also, does it make sense to provide both, just like how we cut blocks?)
Let's defer and keep it simple for now.
yacovm (Mon, 22 Oct 2018 13:45:58 GMT):
So in FAB-12016 it was proposed in the comments to just use the lastest config block of the system channel
yacovm (Mon, 22 Oct 2018 13:46:23 GMT):
but the system channel doesn't record config transactions of channels
yacovm (Mon, 22 Oct 2018 13:47:09 GMT):
and if the consenter nodes completely changed, or the TLS CA cert changed - it would mean that you can't use the system channel to figure out the chains
yacovm (Mon, 22 Oct 2018 13:47:46 GMT):
currently, this problem also exists for peers but at some point we're going to make peers join from the latest config block
yacovm (Mon, 22 Oct 2018 13:47:56 GMT):
and i want to solve this problem for raft OSNs now, if possible
yacovm (Mon, 22 Oct 2018 13:48:12 GMT):
I had proposed a mechanism in the JIRA that involves adding some more information via a config file....
yacovm (Mon, 22 Oct 2018 13:48:19 GMT):
@kostas @jyellick
kostas (Mon, 22 Oct 2018 16:11:25 GMT):
@yacovm: Re-reading that thread now.
kostas (Mon, 22 Oct 2018 16:11:26 GMT):
> We first pull the chains, and pull the system chain last until it reaches the sequence of the bootstrap config block in the file system, which is used to bootstrap the orderer.
kostas (Mon, 22 Oct 2018 16:11:55 GMT):
Let's set aside the concern you raise for now. Remind me this:
kostas (Mon, 22 Oct 2018 16:12:14 GMT):
You feed the new OSN with the latest configuration block of the system channel.
kostas (Mon, 22 Oct 2018 16:12:39 GMT):
The new OSN gets a listing of all the created chains.
kostas (Mon, 22 Oct 2018 16:13:10 GMT):
Does it attempt to connect to all of them, or are we passing in a list of the channels it should reach out to?
kostas (Mon, 22 Oct 2018 16:13:46 GMT):
I am guessing the former, and after a series of unsuccessful attempts, it realizes that it does not have access to a certain chain, and gives up.
kostas (Mon, 22 Oct 2018 16:13:46 GMT):
I am guessing the former, and after a series of unsuccessful attempts, it realizes that it does not have access to a certain chain, and gives up. Correct?
kostas (Mon, 22 Oct 2018 16:13:46 GMT):
I am guessing the former, and after a series of unsuccessful attempts, it realizes that it does not have access to a certain chain, and gives up. Correct? (UPDATE: Yes. Based on your description of how the "cluster" bootstrap method would work.)
kostas (Mon, 22 Oct 2018 16:14:32 GMT):
Now, that's an aside. Just want to make sure I'm on the same page.
kostas (Mon, 22 Oct 2018 16:14:32 GMT):
Now, that's an aside.
kostas (Mon, 22 Oct 2018 16:14:57 GMT):
> and if the consenter nodes completely changed, or the TLS CA cert changed - it would mean that you can't use the system channel to figure out the chains
kostas (Mon, 22 Oct 2018 16:15:00 GMT):
I see the issue.
kostas (Mon, 22 Oct 2018 16:22:09 GMT):
> I had proposed a mechanism in the JIRA that involves adding some more information via a config file....
kostas (Mon, 22 Oct 2018 16:23:12 GMT):
This is tricky as well, no?
kostas (Mon, 22 Oct 2018 16:23:43 GMT):
You need to list in `netboot.yaml` OSNs that can read the system channel, and OSNs that are part of the channels you will join later on.
kostas (Mon, 22 Oct 2018 16:25:03 GMT):
There may be no overlap between the two sets.
kostas (Mon, 22 Oct 2018 16:25:48 GMT):
> Reach out to a majority of OSNs and figure out the latest system channel height and some OSN that possesses it.
kostas (Mon, 22 Oct 2018 16:30:21 GMT):
I am also skeptical about the `majority` bit here. Is it the majority of the OSNs provided in the `netboot.yaml` file? Or is it the majority of the OSNs listed in the provided config block? Both are of limited value from a byzantine perspective.
kostas (Mon, 22 Oct 2018 16:30:21 GMT):
I am also skeptical about the "majority" bit here. Is it the majority of the OSNs provided in the `netboot.yaml` file? Or is it the majority of the OSNs listed in the provided config block? Both are of limited value from a byzantine perspective.
kostas (Mon, 22 Oct 2018 16:54:24 GMT):
As I'm thinking about it more, here's what it comes down to:
kostas (Mon, 22 Oct 2018 16:56:02 GMT):
What is the model, or set of assumptions, that we're adopting? If a consenter set for application channel "foo" can diverge far enough so that it has no overlap with the consenter set recorded in the system channel during foo's creation, then our system channel-based approach is flawed.
kostas (Mon, 22 Oct 2018 16:56:02 GMT):
What is the model, or set of assumptions, that we're adopting? If the consenter set for application channel "foo" can diverge far enough so that it has no overlap with the consenter set recorded in the system channel during foo's creation, then our system channel-based approach is flawed.
kostas (Mon, 22 Oct 2018 16:56:02 GMT):
What is the model, or set of assumptions, that we're adopting? If the consenter set for application channel "foo" can diverge far enough so that it has no overlap with the consenter set recorded in the system channel during foo's creation, then our system channel-based approach is flawed, and neither the "latest system channel config block" or the "cluster"/`netboot.yaml` approach can work.
kostas (Mon, 22 Oct 2018 16:59:52 GMT):
If we assume that at any point in time, only OSNs that are approved by the ordering service of the system channel can be added to application channels, then we're good with the "latest system channel config block" approach that Jason suggested.
kostas (Mon, 22 Oct 2018 17:00:38 GMT):
We can ensure that is the case by setting the modification policies for the cluster set to be restrictive enough and editable by the majority of the ordering service org admins (which is the default I believe).
kostas (Mon, 22 Oct 2018 17:01:08 GMT):
So a channel can go rogue as much as it wants when it comes to adding peer orgs.
kostas (Mon, 22 Oct 2018 17:01:37 GMT):
But when it comes to adding orderer orgs, it is limited to the selection provided to it by the ordering service from which this channel was spawned.
yacovm (Mon, 22 Oct 2018 17:08:06 GMT):
why do you think the netbook approach is flawed ?
yacovm (Mon, 22 Oct 2018 17:08:26 GMT):
all OSNs have system channels in them, no?
yacovm (Mon, 22 Oct 2018 17:08:40 GMT):
even though the system channel might not record these OSNs
yacovm (Mon, 22 Oct 2018 17:09:01 GMT):
they are aware of the system channel heights, since they have the system channels and they are authorized to read them... no?
yacovm (Mon, 22 Oct 2018 17:09:17 GMT):
an OSN is authorized to read a system channel if its org is authorized to read it
yacovm (Mon, 22 Oct 2018 17:09:32 GMT):
and an org of an OSN might be in a system channel, but the OSN itself might not be in it, no?
yacovm (Mon, 22 Oct 2018 17:09:36 GMT):
@kostas
yacovm (Mon, 22 Oct 2018 17:10:00 GMT):
therefore I suggested the net-boot file as a union of the OSNs of all channels that you might need
yacovm (Mon, 22 Oct 2018 17:10:00 GMT):
I suggested the net-boot file as a union of the OSNs of all channels that you might need
yacovm (Mon, 22 Oct 2018 17:10:36 GMT):
essentially - this contains all the communication crypto material + the endpoints that is recorded in the last config block(s) of the channels
yacovm (Mon, 22 Oct 2018 17:10:57 GMT):
and the genesis blocks aside the net-book files, to be able to verify the blocks you pull.
yacovm (Mon, 22 Oct 2018 18:25:30 GMT):
After thinking for a while, I realized that for any OSN - if it's in the system channel, then we know it from the last config block.
If it's not in the system channel - then we can't create a channel via that OSN.... since it's not in the system channel.
yacovm (Mon, 22 Oct 2018 18:25:47 GMT):
but as we know - an OSN can't be without a system channel
yacovm (Mon, 22 Oct 2018 18:26:18 GMT):
so that means that if OSNs diverge from the system channel, it means they actually fork the system channel in a way ;)
yacovm (Mon, 22 Oct 2018 18:26:18 GMT):
so that means that if OSNs diverge from the system channel, it means they actually fork the system channel in a way if they participate in a system channel but it's not the original system channel. But - since you always add nodes incrementally it's not possible to do that
yacovm (Mon, 22 Oct 2018 18:29:28 GMT):
thinking aloud - let's assume that OSNs can diverge from the system channel.
To do that, you need to do a channel reconfiguration that adds a node on a channel and not do a similar reconfiguration in the system channel.
So, the OSN you're adding - is not going to participate in the system channel, and even if it pulls the system chain, it will not participate in it since it will not find its own certificate in it (which currently in our code - we panic if that happens)
yacovm (Mon, 22 Oct 2018 18:31:57 GMT):
Therefore this isn't really a question of "how" but is a question of "what" - what is the right model we want to have in Fabric? Should we even allow adding an OSN to a channel, but not to the system channel?
If we do - then we have the problem that I described above.
If we don't allow this, and all OSNs are always members of the system channel, then joining from the last config block suffices and we don't need the net-book.yaml or anything else....
yacovm (Mon, 22 Oct 2018 18:32:05 GMT):
@kostas @jyellick
yacovm (Mon, 22 Oct 2018 18:39:02 GMT):
The bottom line I'm saying is - given that continue the kafka model where a cluster is only "1" cluster all the time, and OSNs don't "belong" to different clusters - then it makes sense to always add an OSN to the system channel.
Else, if we don't add an OSN to the system channel, we encounter all kinds of problems like inability to create channels with that OSN at all, etc. etc.
yacovm (Mon, 22 Oct 2018 18:56:13 GMT):
now we're left with the "problem" of knowing if an OSN belongs to a chain, or doesn't belong to it. Even if all OSNs belong to the system channel, they might not belong to all chains.
yacovm (Mon, 22 Oct 2018 18:56:50 GMT):
I think the pragmatic approach for now, is to just pull the latest block, look if you're in the chain, and if you're not - then just believe that OSN.
yacovm (Mon, 22 Oct 2018 18:58:19 GMT):
later on in BFT we could just pull the last config block from 2f+1 OSNs, and see the truth.
kostas (Mon, 22 Oct 2018 20:03:41 GMT):
https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=tfzuAv9Cn69j5zXh8
kostas (Mon, 22 Oct 2018 20:03:52 GMT):
Up until this point, are we not saying exactly the same thing?
kostas (Mon, 22 Oct 2018 20:06:17 GMT):
At any rate, my take is this: the only reasonable model we can follow _now_ is one where all OSNs have access to the system channel, i.e. they are endorsed/known/recorded by the cluster that brought up the system channel.
kostas (Mon, 22 Oct 2018 20:07:25 GMT):
I fail to see how we can solve this in a contained manner, if we want to make it so that OSNs can serve an application channel, but don't have access to the system channel.
kostas (Mon, 22 Oct 2018 20:08:07 GMT):
> I think the pragmatic approach for now, is to just pull the latest block, look if you're in the chain, and if you're not - then just believe that OSN.
For instance, I'm not sure if I get what this is proposing.
yacovm (Mon, 22 Oct 2018 20:20:43 GMT):
@kostas - if you have, say - 5 OSNs in the system channel but only 3 of them service channel `foo` - then you don't want all 5 to have channel `foo` in them
kostas (Mon, 22 Oct 2018 21:58:08 GMT):
Alright, so you agree that any application channel can only have a subset of the system channel OSNs as its orderers?
kostas (Mon, 22 Oct 2018 21:58:26 GMT):
(As a pragmatic assumption, given how the code looks today.)
yacovm (Mon, 22 Oct 2018 22:11:47 GMT):
yeah
jyellick (Tue, 23 Oct 2018 01:21:43 GMT):
> mind elaborating a bit?
@guoger The process when an ORDERER_TRANSACTION is encountered (ie a chain creation)
1) "GetOrCreate" the new ledger dir
2) Append the genesis block to the application channel
3) Append the channel creation block to the orderer system channel
If the process is interrupted, then (1) is idempotent, so no harm done. We may detect if (2) has been done if the chain is of length == 1. And (3) completing implies the block was successfully committed.
jyellick (Tue, 23 Oct 2018 01:23:22 GMT):
> At any rate, my take is this: the only reasonable model we can follow _now_ is one where all OSNs have access to the system channel, i.e. they are endorsed/known/recorded by the cluster that brought up the system channel.
+1
guoger (Tue, 23 Oct 2018 03:46:52 GMT):
do we actually rely on the synchronous behavior of `WaitReady` anywhere? what if we just error out if a chain is not ready (not started, stopped, catching up) cc @kostas
guoger (Tue, 23 Oct 2018 03:46:52 GMT):
do we actually (plan to) rely on the synchronous behavior of `WaitReady` anywhere? what if we just error out if a chain is not ready (not started, stopped, catching up) cc @kostas
kostas (Tue, 23 Oct 2018 04:10:55 GMT):
That seems reasonable. Ofc the description on that method should change.
yacovm (Wed, 24 Oct 2018 20:30:10 GMT):
@guoger heads up - I changed https://gerrit.hyperledger.org/r/#/c/27055/ slightly - I removed a parameter from the `newPuller` method - you don't need the consenters anymore, as I made it grab the endpoints straight from the orderer endpoints of the config block.
guoger (Thu, 25 Oct 2018 03:11:00 GMT):
@yacovm are you going to rebase rest of your stack?
yacovm (Thu, 25 Oct 2018 04:55:50 GMT):
Uh... Can you just do it from gerrit @guoger ?
yacovm (Thu, 25 Oct 2018 04:56:09 GMT):
I can do it when i get to the office
guoger (Thu, 25 Oct 2018 05:41:49 GMT):
@yacovm hmmm... is https://gerrit.hyperledger.org/r/c/27173/ supposed to be at the tail of your stack? i might missed something.. could you give it another push when you are available?
guoger (Thu, 25 Oct 2018 05:56:52 GMT):
in [update](https://gerrit.hyperledger.org/r/c/27173/4..5/orderer/common/cluster/replication.go), you were trying to extract a func, however I don't see the extracted func anywhere...
guoger (Thu, 25 Oct 2018 05:56:52 GMT):
in [this update](https://gerrit.hyperledger.org/r/c/27173/4..5/orderer/common/cluster/replication.go), you were trying to extract a func, however I don't see the extracted func anywhere...
yacovm (Thu, 25 Oct 2018 06:16:31 GMT):
It's the end of my stack but you don't need it
yacovm (Thu, 25 Oct 2018 06:16:35 GMT):
you can just use the one below
yacovm (Thu, 25 Oct 2018 06:16:39 GMT):
but you can also rebase on top of it
yacovm (Thu, 25 Oct 2018 06:17:01 GMT):
> I don't see the extracted func anywhere...
I am not using it yet
yacovm (Thu, 25 Oct 2018 06:17:05 GMT):
will use it in the future
cagdast (Thu, 25 Oct 2018 07:50:13 GMT):
Has joined the channel.
guoger (Mon, 29 Oct 2018 12:32:38 GMT):
can someone paste webex link here?
guoger (Mon, 29 Oct 2018 12:34:20 GMT):
are we not having scrum today? @kostas
kostas (Mon, 29 Oct 2018 12:40:45 GMT):
Sorry guys, car trouble.
kostas (Mon, 29 Oct 2018 12:40:59 GMT):
Joining now.
kostas (Mon, 29 Oct 2018 12:41:51 GMT):
@guoger: https://ibm.webex.com/meet/kchrist
kostas (Mon, 29 Oct 2018 12:42:23 GMT):
Yacov and Artem won't be able to attend today, so it's @adarshsaraf123, @guoger, and myself.
john-philipp (Mon, 29 Oct 2018 17:07:32 GMT):
Has joined the channel.
yacovm (Mon, 29 Oct 2018 18:41:32 GMT):
@kostas :
1) Anything interesting in the call?
2) I addressed comments in https://gerrit.hyperledger.org/r/#/c/27138/ , may i get a +2 pretty please?
3) Created https://gerrit.hyperledger.org/r/#/c/27262/ to address the your CR comments to the +2ed / merged CRs
kostas (Mon, 29 Oct 2018 18:51:52 GMT):
@yacovm: 1. No, we didn't end up having it. 2. Done. 3. Thanks!
kostas (Wed, 31 Oct 2018 14:06:46 GMT):
Will need an extra pair of eyes or two @ Adarsh's CR: https://gerrit.hyperledger.org/r/c/27117/ and Artem's CRs: https://gerrit.hyperledger.org/r/c/27238/ https://gerrit.hyperledger.org/r/c/27163/ +/- 1/2 accordingly.
kostas (Wed, 31 Oct 2018 14:06:46 GMT):
Will need an extra pair of eyes or two @ Adarsh's CR: https://gerrit.hyperledger.org/r/c/27117/ and Artem's CRs: https://gerrit.hyperledger.org/r/c/27238/ https://gerrit.hyperledger.org/r/c/27163/ — +/- 1/2 accordingly.
kostas (Wed, 31 Oct 2018 17:50:46 GMT):
https://jira.hyperledger.org/browse/FAB-11937
kostas (Wed, 31 Oct 2018 17:50:46 GMT):
FYI https://jira.hyperledger.org/browse/FAB-11937 - Let's start thinking about the metrics we'd like to expose in the `etcdraft` package. Things that will give us a quick snapshot of what's going in the cluster/channel.
kostas (Thu, 01 Nov 2018 00:48:41 GMT):
Other than a manual check against the acceptance criteria listed above, are we missing anything else before we mark the "multiple channels" story (https://jira.hyperledger.org/browse/FAB-11591) as done?
guoger (Thu, 01 Nov 2018 08:01:56 GMT):
just wanna confirm this: even though already support running user channel on a subset of OSNs, it is not possible w/o type B config, because newly created channels always inherit configurations from system channel, which contains a complete list of OSNs. In another word, we have to _explicitly remove_ an OSN from a user channel, and that requires type B. is this accurate?
guoger (Thu, 01 Nov 2018 08:01:56 GMT):
just wanna confirm this: even though we already support running user channel on a subset of OSNs, it is not possible w/o type B config, because newly created channels always inherit configurations from system channel, which contains a complete list of OSNs. In another word, we have to _explicitly remove_ an OSN from a user channel, and that requires type B. is this accurate?
yacovm (Thu, 01 Nov 2018 08:16:44 GMT):
that's a good point :(
yacovm (Thu, 01 Nov 2018 08:16:57 GMT):
@jyellick @kostas your opinion?
kostas (Thu, 01 Nov 2018 12:26:03 GMT):
I think this is accurate. And this may make the whole "each channel on a different subset" idea a bit problematic.
kostas (Thu, 01 Nov 2018 12:27:52 GMT):
Let's see if Jason confirms that this is indeed the case, and if it is, whether there's a way to alter the channel creation path so that we can achieve what we want.
jyellick (Thu, 01 Nov 2018 14:11:58 GMT):
@kostas @guoger I think I'm missing what "type B config" means.
kostas (Thu, 01 Nov 2018 14:12:46 GMT):
A configuration update that modifies the consenters.
kostas (Thu, 01 Nov 2018 14:12:51 GMT):
(Essentially.)
jyellick (Thu, 01 Nov 2018 14:13:08 GMT):
I would however indicate/mention that it is entirely possible to construct a new channel with a custom set of orderers. Yes, the ephemeral configuration which is the precursor to the genesis block inherits everything from the orderer system channel, but the 'channel creation tx' is an channel config update against that ephemeral config, and you may modify any elements which are mutable (including the orderer addresses).
kostas (Thu, 01 Nov 2018 14:13:44 GMT):
Ah, that is great.
kostas (Thu, 01 Nov 2018 14:13:58 GMT):
That means we can have a system channel with A, B, C, D, E as its consenters.
kostas (Thu, 01 Nov 2018 14:14:11 GMT):
And channel foo is created with A, C, E only correct?
kostas (Thu, 01 Nov 2018 14:14:31 GMT):
(It's a modification over what the system channel consenter set reads.)
jyellick (Thu, 01 Nov 2018 14:15:11 GMT):
Correct. However, I would note, by default, to modify the set of orderers/consenters will require that there be signatures included on the creation tx which satisfy the /Channel/Orderer/Admins policy (with a single org, this would be the signature of an orderer admin).
jyellick (Thu, 01 Nov 2018 14:15:11 GMT):
Correct. However, I would note, by default, to modify the set of orderers/consenters will require that there be signatures included on the creation tx which satisfy the /Channel/Orderer/Admins policy (with a single orderer org, this would be the signature of an orderer admin).
C0rWin (Thu, 01 Nov 2018 14:46:54 GMT):
@kostas @jyellick so, just to confirm. if we create a new user channel with consenters set which is a subset of consenters in system channel, at this tx signature conforms `/Channel/Orderer/Admins` policy new channel configuration will be updated to include only declared subset.. am I right?
I.e. if system channel contains as @kostas mentioned A, B, C, D, E and new channel declares to have only B, C, E signed with orderer admin the consenters set will be automatically updated?
jyellick (Thu, 01 Nov 2018 14:48:28 GMT):
Correct. The config in the genesis block of the new channel is the post-image of applying the config update to the ephemeral config. The ephemeral config contains all consenters, the config update may modify them.
john-philipp (Thu, 01 Nov 2018 15:49:24 GMT):
Let me know if this isn't the correct channel to ask this is:
Reading the design document you reemphasise to only ever edit one consenting node at a time, which makes sense, however, what happens when in a two-node system another consenter is registered that then never starts voting (assuming 2f + 1)?
Is there a protocol level defense against this? Or is it simply a matter of any network is allowed to kill itself?
kostas (Thu, 01 Nov 2018 17:15:07 GMT):
@john-philipp: There is no protocol level defense against this because that node would be byzantine.
kostas (Thu, 01 Nov 2018 17:15:49 GMT):
For the particular example that you describe, we'd still be able to operate with just 2 active nodes, so long as that 3rd node was idle and wouldn't cast bad votes.
kostas (Thu, 01 Nov 2018 17:18:42 GMT):
If we were to re-examine your question in a BFT setting, I'd say the answer is more or less still the same. That new node can do whatever it wants (abstain, cast bad votes, whatever) and as long as we have up to f of those nodes in, we're good. Anything beyond that, and the protocol cannot defend against it.
kostas (Thu, 01 Nov 2018 17:18:51 GMT):
Let me know if that answers the question.
john-philipp (Fri, 02 Nov 2018 09:57:16 GMT):
@kostas: Well, sort of. :)
Clearly I am new here, so feel free to point out any potential lack of understanding. I've put some effort into reading your design, but of course I may have missed something.
In a BFT setting we want to protect ourselves from attack surfaces requiring less than *f* players. In the two node example (assuming *2f + 1*) the third player wouldn't be able to do whatever he wants, in fact he absolutely has to play by the rules, since otherwise *f = 1*, and we'd require * n_honest >= 2(1) + 1 = 3*. But *n_honest = 2*. //
Which hopefully leads to a clearer definition of my question:
How are you planning to protect yourself against malicious voter creation, that is one node suggesting new voters that may not exist. And even if most nodes would have to agree to such a config change, specifically how would/could they?
Now this seems like a high-level problem, but it has the potential to completely circumvent any BFT mechanisms, so I was just wondering how you are planning to address it.
yacovm (Fri, 02 Nov 2018 10:00:41 GMT):
the config change is enforced via signature checks of majority of organizations among the orderer orgs @john-philipp
yacovm (Fri, 02 Nov 2018 10:00:49 GMT):
a majority of 2 is just 2....
yacovm (Fri, 02 Nov 2018 10:01:10 GMT):
and in case of BFT you have to start with 4
yacovm (Fri, 02 Nov 2018 10:01:25 GMT):
you can't start with 2
john-philipp (Fri, 02 Nov 2018 11:48:51 GMT):
@yacovm : Appreciate the response. I think we're getting closer.
_n=3_ was really only a small example without loss of generality.
I am interested in the following:
* Who is _allowed to propose_ a consenter addition to the existing set?
* Based on what rules do orderer organisations _sign-off_ on a config change that introduces a new consenter?
john-philipp (Fri, 02 Nov 2018 11:50:07 GMT):
By propose I mean submit config tx to the ordering cluster (... I think).
yacovm (Fri, 02 Nov 2018 13:16:34 GMT):
the admins of the existing orgs are allowed to propose a consenter addition
yacovm (Fri, 02 Nov 2018 13:16:45 GMT):
the rules are flexible.... it's just a policy
john-philipp (Fri, 02 Nov 2018 13:51:24 GMT):
Right, I am catching up. :) There is an out-of-band communication I didn't expect. So basically, new consenter endorsement is a _manual_ step which isn't communicated through the ledger itself. I understand this better now, thanks.
guoger (Fri, 02 Nov 2018 15:14:44 GMT):
i don't understand why this test is failing...
```
Running Suite: Grpclogging Suite
================================
Random Seed: 1541170356
Will run 23 of 23 specs
•••••
------------------------------
• Failure [0.820 seconds]
Server
/w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/common/grpclogging/server_test.go:33
UnaryServerInterceptor
/w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/common/grpclogging/server_test.go:114
logs request data [It]
/w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/common/grpclogging/server_test.go:130
Expected
yacovm (Sun, 04 Nov 2018 20:21:45 GMT):
Looks like there is a flaky test ?
https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/5802/console
guoger (Mon, 05 Nov 2018 02:33:32 GMT):
it should be fixed by Artem's [later CR](https://gerrit.hyperledger.org/r/c/27350/4/orderer/consensus/etcdraft/chain.go#924), although maybe worth to be migrated to the [first one](https://gerrit.hyperledger.org/r/c/27163/29/orderer/consensus/etcdraft/chain.go#907)
guoger (Mon, 05 Nov 2018 03:46:17 GMT):
IIUC, `ORDERER_GENERAL_TLS_ENABLED` becomes mandatory in etcdraft? cc @yacovm
yacovm (Mon, 05 Nov 2018 10:32:12 GMT):
yep @guoger
yacovm (Mon, 05 Nov 2018 11:35:16 GMT):
@guoger @C0rWin @kostas - when we create a new channel with a subset of OSNs (a subset of the system channel OSNs) - do we have code that makes an OSN that isn't in the channel, to skip adding the genesis block to its ledger?
yacovm (Mon, 05 Nov 2018 11:36:00 GMT):
(I'm asking because it's easier to save my current "dirty" git repo and pull the latest change set and see ;) )
yacovm (Mon, 05 Nov 2018 11:36:00 GMT):
(I'm asking because it's easier to ask than to save my current "dirty" git repo and pull the latest change set and see ;) )
yacovm (Mon, 05 Nov 2018 11:36:28 GMT):
If we don't have such code we'll need it... I think.
yacovm (Mon, 05 Nov 2018 11:36:47 GMT):
and we'll also need to handle an addition of a node to a channel, when that node is already in the system channel
yacovm (Mon, 05 Nov 2018 11:36:48 GMT):
no?
guoger (Mon, 05 Nov 2018 11:36:59 GMT):
i don't think so... when a new channel is created, our code makes a copy of system channel config
guoger (Mon, 05 Nov 2018 11:36:59 GMT):
i don't think so... when a new channel is created, our code makes a copy of system channel config while processing the config update, and assemble the new config block (genesis block)
yacovm (Mon, 05 Nov 2018 11:37:02 GMT):
of course, 1 thing at a time
yacovm (Mon, 05 Nov 2018 11:37:07 GMT):
just thinking out loud
kostas (Mon, 05 Nov 2018 13:31:38 GMT):
Anybody else have trouble joining the meeting?
kostas (Mon, 05 Nov 2018 13:31:44 GMT):
draggingScreenshot.png
kostas (Mon, 05 Nov 2018 13:32:17 GMT):
Let's switch to Zoom if you don't mind: https://zoom.us/j/7432937602
kostas (Mon, 05 Nov 2018 13:32:36 GMT):
@adarshsaraf123 @yacovm @guoger @C0rWin
kostas (Mon, 05 Nov 2018 13:34:09 GMT):
Nevermind, we got WebEx working again, so we're switching there.
guoger (Mon, 05 Nov 2018 13:38:27 GMT):
joining
kostas (Mon, 05 Nov 2018 13:49:13 GMT):
I forgot to say: when you add functional tests, please update the description in https://jira.hyperledger.org/browse/FAB-6135
kostas (Mon, 05 Nov 2018 13:49:29 GMT):
draggingScreenshot.png
kostas (Mon, 05 Nov 2018 13:51:24 GMT):
> when we create a new channel with a subset of OSNs (a subset of the system channel OSNs) - do we have code that makes an OSN that isn't in the channel, to skip adding the genesis block to its ledger?
kostas (Mon, 05 Nov 2018 13:52:21 GMT):
@yacovm: Good catch.
guoger (Mon, 05 Nov 2018 13:54:54 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=yz93a8YAYNYMBxz9c) just to make sure IIUC, @yacovm is referring to this topic, correct?
yacovm (Mon, 05 Nov 2018 13:57:12 GMT):
I am not really referring to that
yacovm (Mon, 05 Nov 2018 13:57:17 GMT):
you can have a custom config
yacovm (Mon, 05 Nov 2018 13:57:41 GMT):
but it won't make the OSN that its certificate is *not* in the config *skip* from participating in the consensus
yacovm (Mon, 05 Nov 2018 13:57:46 GMT):
and initializing the chain object....
yacovm (Mon, 05 Nov 2018 13:58:13 GMT):
as i said it's not blocking me *now* because i want to walk before I run :)
yacovm (Mon, 05 Nov 2018 13:58:29 GMT):
but we need to add some code... I'll open a JIRA for that
yacovm (Mon, 05 Nov 2018 13:58:39 GMT):
A "story" JIRA - @kostas , don't worry ;)
kostas (Mon, 05 Nov 2018 13:58:56 GMT):
Not a story! I'm gonna have a heart attack.
yacovm (Mon, 05 Nov 2018 13:59:05 GMT):
you said we need a story for those no?
yacovm (Mon, 05 Nov 2018 13:59:09 GMT):
:thinking:
kostas (Mon, 05 Nov 2018 13:59:23 GMT):
All of these discoveries should be filed as sub-tasks under a story around "onboarding".
yacovm (Mon, 05 Nov 2018 13:59:31 GMT):
no but this is not a task, this is a story
yacovm (Mon, 05 Nov 2018 13:59:38 GMT):
this is not related to onboarding IMO
yacovm (Mon, 05 Nov 2018 13:59:54 GMT):
this is related to reconfiguration
yacovm (Mon, 05 Nov 2018 13:59:56 GMT):
no?
kostas (Mon, 05 Nov 2018 14:00:18 GMT):
You are right actually. Reconfiguration.
guoger (Mon, 05 Nov 2018 14:09:39 GMT):
if it couldn't detect its own raft id, doesn't it fail to create chain object?
guoger (Mon, 05 Nov 2018 14:09:39 GMT):
if it couldn't detect its own raft id, doesn't it fail to create chain object?
> if it couldn't detect its own raft id, doesn't it fail to create chain object?
guoger (Mon, 05 Nov 2018 14:09:39 GMT):
if it couldn't detect its own raft id, doesn't it fail to create chain object?
>but it won't make the OSN that its certificate is *not* in the config *skip* from participating in the consensus
guoger (Mon, 05 Nov 2018 14:09:39 GMT):
if it couldn't detect its own raft id, doesn't it fail to create chain object? cc @yacovm
>but it won't make the OSN that its certificate is *not* in the config *skip* from participating in the consensus
yacovm (Mon, 05 Nov 2018 14:15:02 GMT):
that's part of the problem @guoger
yacovm (Mon, 05 Nov 2018 14:15:04 GMT):
it panics....
yacovm (Mon, 05 Nov 2018 14:15:09 GMT):
but we don't want to get there
yacovm (Mon, 05 Nov 2018 14:15:18 GMT):
we don't want the chain support object to be created
yacovm (Mon, 05 Nov 2018 14:24:19 GMT):
I'll open a JIRA today and tag the relevant people
kostas (Mon, 05 Nov 2018 19:31:15 GMT):
@tock: Can you expand a bit on the set of scripts you're working on for those tests against the acceptance criteria? Artem touched on it briefly during our scrum today.
tock (Mon, 05 Nov 2018 19:31:15 GMT):
Has joined the channel.
guoger (Tue, 06 Nov 2018 02:28:11 GMT):
as I was replying to @jyellick in a CR comment, and I think this point is worth to be called out here:
> we should not suggest users to alter raft-related config options (at least for now), i.e. TickInterval, HeartbeatTimeout, ElectionTimeout, etc
They are put in channel config to enforce consistency across a channel, because difference of ticks would result in unstable cluster otherwise. However, we could *not* alter them while node is up and running, due to the limitation of etcd/raft lib. Change of these configs will take effect on next boot, however we end up with an unstable cluster if those nodes are not rebooted synchronously
guoger (Tue, 06 Nov 2018 02:29:03 GMT):
cc @kostas should we explicitly check against those configurations?
awes0menessInc (Tue, 06 Nov 2018 03:39:09 GMT):
Has joined the channel.
tock (Tue, 06 Nov 2018 07:18:04 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=2aFHquoGS4MSCbXZG) @kostas First, I am augmenting the fabric-samples/first-network to include an etcd/raft orderer option. I will keep this sample "simple" and support only a single orderer. Next, I plan to take this sample and turn it into a more extensive example: 3 raft orderers, multiple channels, injecting transactions, stopping an orderer, injecting transactions, restarting an orderer, checking that it caught up. Not sure that this fits to the scope of fabric-samples thought. We'll have to think where to put it. Right now in remains in my private repo until further decision.
tock (Tue, 06 Nov 2018 07:18:04 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=2aFHquoGS4MSCbXZG) @kostas First, I am augmenting the fabric-samples/first-network to include an etcd/raft orderer option. I will keep this sample "simple" and support only a single orderer. Next, I plan to take this sample and turn it into a more extensive example/test: 3 raft orderers, multiple channels, injecting transactions, stopping an orderer, injecting transactions, restarting an orderer, checking that it caught up. Not sure that this fits to the scope of fabric-samples thought. We'll have to think where to put it. Right now in remains in my private repo until further decision.
guoger (Tue, 06 Nov 2018 07:53:53 GMT):
@tock maybe you wanna take a look at these two CRs? I think they captured part of the scenario you described? https://gerrit.hyperledger.org/r/c/27249/7 & https://gerrit.hyperledger.org/r/c/27330/3
yacovm (Tue, 06 Nov 2018 08:16:57 GMT):
@guoger regarding altering the ticks, etc. - i think for now it's good enough documenting that users not do this without off-band collaboration and proper maintenance
kisna (Tue, 06 Nov 2018 18:48:46 GMT):
Has joined the channel.
kisna (Tue, 06 Nov 2018 18:50:10 GMT):
hello, trying to customize an e2e test in Fabric 1.3-rc1 but orderer keeps failing
orderer still cannot find e2e-orderer-syschan?
and cli keeps searching for Attempting to fetch system channel 'e2e-orderer-syschan' ..XXX secs
or an orderer detailed setup/documentation?
kisna (Tue, 06 Nov 2018 18:50:58 GMT):
did re generate crypto keys certs and the tx blocks successfully, also inspected them that all domains are correctly updated
question is, how do I verify that the default channels like e2e-orderer-syschan are created in the orderer?
`../../.build/bin/cryptogen generate --config=crypto-config.yaml
../../.build/bin/configtxgen -profile TwoOrgsOrdererGenesis -outputBlock ./channel-artifacts/genesis.block
export CHANNEL_NAME=custom_domain_channel && ../../.build/bin/configtxgen -profile TwoOrgsChannel -outputCreateChannelTx ./channel-artifacts/channel.tx -channelID $CHANNEL_NAME
.. anchor peer .tx files
`
../../.build/bin/configtxgen -channelID custom_channel -outputBlock custom_channel_genesisblock.pb -inspectBlock custom_channel_genesisblock.pb -profile TwoOrgsOrdererGenesis
also returns all MSPs, orderers properly, looks like orderer is not coming up fine, *i.e., their system channels are not and I don't know how to investigate this*
kisna (Tue, 06 Nov 2018 18:50:58 GMT):
did re generate crypto keys certs and the tx blocks successfully, also inspected them that all domains are correctly updated
question is, how do I verify that the default channels like e2e-orderer-syschan are created in the orderer?
`../../.build/bin/cryptogen generate --config=crypto-config.yaml
../../.build/bin/configtxgen -profile TwoOrgsOrdererGenesis -outputBlock ./channel-artifacts/genesis.block
export CHANNEL_NAME=custom_domain_channel && ../../.build/bin/configtxgen -profile TwoOrgsChannel -outputCreateChannelTx ./channel-artifacts/channel.tx -channelID $CHANNEL_NAME
.. anchor peer .tx files
`
../../.build/bin/configtxgen -channelID custom_channel -outputBlock custom_channel_genesisblock.pb -inspectBlock custom_channel_genesisblock.pb -profile TwoOrgsOrdererGenesis
also returns all MSPs, orderers properly, looks like orderer is not coming up fine, *i.e., their system channels are not and I don't know how to investigate this*
kisna (Tue, 06 Nov 2018 18:50:58 GMT):
did re generate crypto keys certs and the tx blocks successfully, also inspected them that all domains are correctly updated
question is, how do I verify that the default channels like e2e-orderer-syschan are created in the orderer?
`../../.build/bin/cryptogen generate --config=crypto-config.yaml
../../.build/bin/configtxgen -profile TwoOrgsOrdererGenesis -outputBlock ./channel-artifacts/genesis.block
export CHANNEL_NAME=custom_domain_channel && ../../.build/bin/configtxgen -profile TwoOrgsChannel -outputCreateChannelTx ./channel-artifacts/channel.tx -channelID $CHANNEL_NAME
.. anchor peer .tx files
`
../../.build/bin/configtxgen -channelID custom_channel -outputBlock custom_channel_genesisblock.pb -inspectBlock custom_channel_genesisblock.pb -profile TwoOrgsOrdererGenesis
also returns all MSPs, orderers properly, looks like orderer is not coming up fine, *i.e., their system channels are not and I don't know how to investigate this*
kostas (Tue, 06 Nov 2018 19:21:04 GMT):
kisna
kostas (Tue, 06 Nov 2018 19:24:47 GMT):
When you get all get chance, do give a look at Adarsh's and Artem's CRs please.
kisna (Tue, 06 Nov 2018 19:49:25 GMT):
Has left the channel.
kostas (Wed, 07 Nov 2018 02:01:55 GMT):
Raft-related integration test failure on this one: https://gerrit.hyperledger.org/r/c/27117/
kostas (Wed, 07 Nov 2018 02:01:58 GMT):
``` [Fail] EndToEnd Crash Fault Tolerance when orderer stops and restarts [It] keeps network up and running
17:44:51 /w/workspace/fabric-verify-integration-tests-x86_64/gopath/src/github.com/hyperledger/fabric/integration/e2e/e2e_test.go:318
17:44:51
17:44:51 [Fail] EndToEnd basic single node etcdraft network with 2 orgs and 2 channels [It] executes a basic etcdraft network with 2 orgs and 2 channels ```
adarshsaraf123 (Wed, 07 Nov 2018 08:01:51 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=H669emNSe4Xso2q7R) @kostas Working on it. Can't nail down the cause yet.
enriquebusti (Wed, 07 Nov 2018 12:01:39 GMT):
Has joined the channel.
C0rWin (Wed, 07 Nov 2018 17:00:01 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=sno42hHksye7xf5Dh) @kostas I've reviewed @adrash work and have few concerns, provided my comments in CR.
kostas (Wed, 07 Nov 2018 19:23:14 GMT):
@C0rWin: Thanks for taking a look! Fairly certain that the first comment is on making that channel instead of draining it; not the parent method. As for the other code block - Adarsh and I were discussing about it earlier today before the review; this is how we prevent ourselves from reading from `submitC`. Let's get Adarsh to chime in on the CR though — perhaps there is something that we've missed there.
kostas (Wed, 07 Nov 2018 19:23:14 GMT):
@C0rWin: Thanks for taking a look! ~Fairly~ Reasonably certain that the first comment is on making that channel instead of draining it; not the parent method. As for the other code block - Adarsh and I were discussing about it earlier today before the review; this is how we prevent ourselves from reading from `submitC`. Let's get Adarsh to chime in on the CR though — perhaps there is something that we've missed there.
kostas (Wed, 07 Nov 2018 19:23:14 GMT):
@C0rWin: Thanks for taking a look! ~Fairly~ Reasonably certain that the first comment is on making that channel instead of draining it; not the parent method. As for the other code block, that gave me pause as well and in fact Adarsh and I were discussing about it earlier today before the review; this is how we prevent ourselves from reading from `submitC`. Let's get Adarsh to chime in on the CR though — perhaps there is something that we've missed there.
C0rWin (Wed, 07 Nov 2018 23:29:35 GMT):
> his is how we prevent ourselves from reading from `submitC`
I'm not following here, didn't we decouple next block and block commit to being able to parallelize consenting on block and block commit at first place?
adarshsaraf123 (Thu, 08 Nov 2018 04:31:52 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=PggP6HKCEMuA3Zttv) @kostas Thanks Kostas. The first one is indeed a comment on making that channel instead of draining it. I left the comment in since earlier the same concern had been raised.
adarshsaraf123 (Thu, 08 Nov 2018 04:41:37 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=PACsTQbD9ZyszP2HY) @C0rWin So the thing is that we can only pipeline the creation and writing of normal blocks. When it comes to config blocks, we must wait for the write of the config block (which also leads to the config change being applied). If we do not do this we end up processing the future txns at an older config sequence. To clarify this further, consider the following scenario:
1. Current `batchsize` is 20.
adarshsaraf123 (Thu, 08 Nov 2018 04:41:37 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=PACsTQbD9ZyszP2HY) @C0rWin So the thing is that we can only pipeline the creation and writing of normal blocks. When it comes to config blocks, we must wait for the write of the config block (which also leads to the config change being applied). If we do not do this we end up processing the future txns at an older config sequence. To clarify this further, consider the following scenario:
1. Current `batchsize` is 20.
2. A config txn that changes the `batchsize` to 10 arrives and we create the config block and do not wait for its write.
3. We create the next normal block with 20 txns since the config change has been applied.
In this scenario, if we look at the ledger, then the normal block after the config block is incorrect since it does not adhere to the new configuration.
adarshsaraf123 (Thu, 08 Nov 2018 06:15:59 GMT):
@C0rWin Just curious about what motivated this CR
https://gerrit.hyperledger.org/r/#/c/27372/
guoger (Thu, 08 Nov 2018 06:27:02 GMT):
@adarshsaraf123 that is to mitigate flaky tests in case of slow IO
adarshsaraf123 (Thu, 08 Nov 2018 06:27:31 GMT):
Thanks @guoger
C0rWin (Thu, 08 Nov 2018 09:14:34 GMT):
@adarshsaraf123 @guoger thanks for explanations. though I think this needs to be rebased at first place regardless to comments otherwise it will be conflicting with rest of the stack of commits we have out there.
adarshsaraf123 (Thu, 08 Nov 2018 09:20:14 GMT):
Sure @C0rWin I will work on the rebasing later in the day. I saw that [this CR](https://gerrit.hyperledger.org/r/#/c/27350/) had a -1 in VerifyBuild and that is why held out the rebasing for future. I will do the rebasing once the CRs a bit stable.
adarshsaraf123 (Thu, 08 Nov 2018 09:20:14 GMT):
Sure @C0rWin I will work on the rebasing later in the day. I saw that [this CR](https://gerrit.hyperledger.org/r/#/c/27350/) had a -1 in VerifyBuild and that is why held out the rebasing for future. I will do the rebasing once the CRs are a bit stable.
kostas (Thu, 08 Nov 2018 15:14:11 GMT):
As best as I can tell, everything up until this point is merge-able: https://gerrit.hyperledger.org/r/c/27349/
kostas (Thu, 08 Nov 2018 15:14:39 GMT):
If you disagree, as they say in American weddings in the movies, speak now or forever hold your silence.
kostas (Thu, 08 Nov 2018 15:14:56 GMT):
(Otherwise, let's get that pending stack down.)
C0rWin (Thu, 08 Nov 2018 15:17:56 GMT):
I will review now the snapshotting part
C0rWin (Thu, 08 Nov 2018 15:18:18 GMT):
just done w/ first one in a stack (re: WAL)
C0rWin (Thu, 08 Nov 2018 15:18:59 GMT):
and we can always address comments/concerns post factum
C0rWin (Thu, 08 Nov 2018 15:20:10 GMT):
IMO, it's better to have it in to reduce overhead of perpetual daily rebasing
kostas (Thu, 08 Nov 2018 15:20:45 GMT):
I concur. We've been more-or-less rolling like this for a few weeks now.
C0rWin (Thu, 08 Nov 2018 15:20:49 GMT):
of course unless there is major issue
adarshsaraf123 (Thu, 08 Nov 2018 20:05:34 GMT):
I am also in the midst of reviewing Artem's type B config changes. Can we please defer that for merging?
adarshsaraf123 (Thu, 08 Nov 2018 20:06:06 GMT):
@kostas ^^
kostas (Thu, 08 Nov 2018 20:06:26 GMT):
Sure thing. Thanks for taking a look!
kostas (Mon, 12 Nov 2018 14:51:27 GMT):
Whoops: https://twitter.com/copyconstruct/status/1061818753925578753
kostas (Mon, 12 Nov 2018 14:51:47 GMT):
It'll be fascinating to see whether we agree with this realization a few months down the line as well.
C0rWin (Mon, 12 Nov 2018 22:10:01 GMT):
Well, we are not implementing Raft neither Paxos, we just integrating it into Fabric. But in general that’s true, the fact that it’s simplier for human to understand doesn’t mean it’s simply to implement.
kostas (Mon, 12 Nov 2018 22:20:06 GMT):
CockroachDB is not implementing Raft either. They're using etcd/raft.
C0rWin (Tue, 13 Nov 2018 07:20:06 GMT):
oh, from this perspective we only started :) ... there are many interesting Raft specific revelations underway, I'm sure...
john-philipp (Wed, 14 Nov 2018 11:09:07 GMT):
Heya, I was hoping to debug the ordering service to get a better understanding of the internal message flow. Is there some sort of trick? I tried in goland and vscode. Error is .debug_frame section missing. Any pointers?
Exec summary:
* I can't debug fabric.
* How do I debug fabric? :)
john-philipp (Wed, 14 Nov 2018 11:09:07 GMT):
Heya, I was hoping to debug the ordering service to get a better understanding of the internal message flow. Is there some sort of trick? I tried in goland and vscode. Error is .debug_frame section missing. Any pointers?
Exec summary:
* I can't debug the ordering service.
* How do I debug the ordering service? :)
jyellick (Wed, 14 Nov 2018 15:57:16 GMT):
@john-philipp Because fabric uses golang plugins the debuggers can have some troubles. My recollection is that the development version of go v1.12 may get you past some of these, but I no promises. In general, I've found go debuggers to be of limited value.
john-philipp (Wed, 14 Nov 2018 16:03:19 GMT):
@jyellick Yes, I am trying to installing 1.12 now, let's see how that goes. Thanks.
yacovm (Wed, 14 Nov 2018 16:09:28 GMT):
you mean debug as in use the debugger? it's multi-threaded and distributed (i.e - etcdraft) - not exactly the ideal code to use a debugger on
yacovm (Wed, 14 Nov 2018 16:10:07 GMT):
maybe you mean to debug a specific section or function?
john-philipp (Wed, 14 Nov 2018 16:15:14 GMT):
@yacovm I was hoping to use a debugger yes. Specific sections or functions would be acceptable. I come from Python I am beginning to understand that I may have been spoiled. :/
Basically, I am looking to set up an effective dev environment. But by the above, I do have to ask, what *do* you use? Do you debug at all?
yacovm (Wed, 14 Nov 2018 16:15:31 GMT):
I use `fmt.Println`
john-philipp (Wed, 14 Nov 2018 16:15:53 GMT):
... I see. I was hoping to avoid as much.
john-philipp (Wed, 14 Nov 2018 16:34:00 GMT):
Incidentally, I do seem to have some debugging capability on the ordering service now. :]
magar36 (Thu, 15 Nov 2018 17:28:36 GMT):
Has joined the channel.
magar36 (Thu, 15 Nov 2018 17:48:49 GMT):
Hello Everyone!
I have been trying to understand the way grpc over http2 works under the hood in fabric. What I am trying to find out is how exactly does the orderer disseminate the block to the leader peers and I am talking from networking perspective.
So we know that there is multiplexing supported in http2. In other words, the client (say peer) uses a source port to connect to the destination port (say orderer). Each source and destination port pairing identifies a separate virtual connection, allowing multiple connections to share one physical network connection - meaning multiple parallel requests can be sent over the same persistent connection. Now that we have the physical connection established, how does the destination send any information back to the source that is not the part of the request? Mind that the block that gets sent by the orderer to the peers is not a server push mechanism and it gets sent only when the block is ready to be sent. In other words, it is not something that orderer sends to the peers as part of a request.
I have taken balance-transfer as an example from the fabric-samples project to take the packet capture and analyze it through wireshark. Since the entire communication uses grpcs, I am unable to get the exact information as to when is the block being sent by the orderer and corresponding header/data details to understand more.
Is there a way that the balance-transfer example can be modified to use grpc instead of grpcs so packet capture can make things much easier?
magar36 (Thu, 15 Nov 2018 17:49:38 GMT):
I do hope that the question makes some sense to atleast someone or else let me know. I would really appreciate any ideas here.
jyellick (Thu, 15 Nov 2018 17:50:14 GMT):
@magar36 This is a better question for #fabric-orderer I will cross-post it there for you.
magar36 (Thu, 15 Nov 2018 17:51:30 GMT):
thank you
Skprog (Fri, 16 Nov 2018 03:12:27 GMT):
Has joined the channel.
BellaAdams (Sat, 17 Nov 2018 00:46:06 GMT):
Has joined the channel.
guoger (Mon, 19 Nov 2018 04:23:22 GMT):
in etcdraft, `Step` is a super frequent unary grpc call, and with [commit I868f1e13](https://github.com/hyperledger/fabric/commit/91e0b0b9dde01b52914e13ed1375ad1359fdc30b), log is overwhelmed with `step` messages. is it possible to suppress an interceptor for a particular api? cc @sykesm @kostas
guoger (Mon, 19 Nov 2018 13:30:44 GMT):
webex is still loading...
kostas (Mon, 19 Nov 2018 15:12:10 GMT):
Let me look. Thanks for the heads up.
huxiangdong (Tue, 20 Nov 2018 00:33:46 GMT):
Has joined the channel.
tock (Thu, 22 Nov 2018 13:31:13 GMT):
I assume no sync today... Happy thanksgiving :-)
kostas (Thu, 22 Nov 2018 14:23:31 GMT):
Correct, no participation from me anyway, as noted in Monday's scrum. And thanks!
Unni_1994 (Fri, 23 Nov 2018 10:31:04 GMT):
Has joined the channel.
kostas (Mon, 26 Nov 2018 19:14:09 GMT):
https://gerrit.hyperledger.org/r/c/27163/
kostas (Mon, 26 Nov 2018 19:14:12 GMT):
https://gerrit.hyperledger.org/r/c/27350/
kostas (Mon, 26 Nov 2018 19:15:13 GMT):
These ones are missing a +2 and they'll allow us to get 8 CRs in today.
kostas (Mon, 26 Nov 2018 19:15:13 GMT):
These ones are missing a +2 and they'll allow us to get 8 CRs in.
kostas (Mon, 26 Nov 2018 19:15:30 GMT):
I'd ask in #fabric-pr-review but they require domain knowledge to get in.
kostas (Mon, 26 Nov 2018 19:16:13 GMT):
@yacovm I know you're swamped, but have a look when you get a chance? (These are Artem's CRs so he can't +2.)
kostas (Mon, 26 Nov 2018 19:16:13 GMT):
@yacovm I know you're swamped, but have a look whenever you get a chance? (These are Artem's CRs so he can't +2.)
yacovm (Mon, 26 Nov 2018 19:34:31 GMT):
I can take a look tomorrow :)
yacovm (Mon, 26 Nov 2018 19:34:43 GMT):
oh hmmm
yacovm (Mon, 26 Nov 2018 19:48:47 GMT):
I merged some. will CR the 2nd one tomorrow
kostas (Tue, 27 Nov 2018 01:32:16 GMT):
Color me confused on this one.
kostas (Tue, 27 Nov 2018 01:32:24 GMT):
The Viper documentation reads:
kostas (Tue, 27 Nov 2018 01:32:47 GMT):
```Viper uses the following precedence order. Each item takes precedence over the item below it:
- explicit call to Set
- flag
- env
- config
- key/value store
- default```
kostas (Tue, 27 Nov 2018 01:33:12 GMT):
Notice how env takes precedence over config.
kostas (Tue, 27 Nov 2018 01:33:39 GMT):
It is my expectation then, that:
kostas (Tue, 27 Nov 2018 01:34:23 GMT):
1. If I remove, say, the `Consensus` section from the `orderer.yaml`: https://github.com/hyperledger/fabric/blob/69a34181ce4cb8eb890509ec9de3dbb8e77df6e4/sampleconfig/orderer.yaml#L311..L321
kostas (Tue, 27 Nov 2018 01:34:23 GMT):
1. If I remove, say, the `Cluster` section from `orderer.yaml`: https://github.com/hyperledger/fabric/blob/master/sampleconfig/orderer.yaml#L54..L79
kostas (Tue, 27 Nov 2018 01:35:42 GMT):
2. But keep the backing data structure: https://github.com/hyperledger/fabric/blob/69a34181ce4cb8eb890509ec9de3dbb8e77df6e4/orderer/common/localconfig/config.go#L46
kostas (Tue, 27 Nov 2018 01:36:32 GMT):
3. And set those configuration options via ENV vars: https://github.com/hyperledger/fabric/blob/69a34181ce4cb8eb890509ec9de3dbb8e77df6e4/orderer/common/server/etcdraft_test.go#L80..L82
kostas (Tue, 27 Nov 2018 01:36:32 GMT):
3. And set those configuration options via environment variables: https://github.com/hyperledger/fabric/blob/69a34181ce4cb8eb890509ec9de3dbb8e77df6e4/orderer/common/server/etcdraft_test.go#L80..L82
kostas (Tue, 27 Nov 2018 01:38:56 GMT):
I should still expect the `Cluster` field in the backing data structure to be populated with the ENV var values when loading the configuration: https://github.com/hyperledger/fabric/blob/69a34181ce4cb8eb890509ec9de3dbb8e77df6e4/orderer/common/server/main.go#L78
kostas (Tue, 27 Nov 2018 01:39:02 GMT):
However, that is not the case.
kostas (Tue, 27 Nov 2018 01:42:07 GMT):
All `Cluster` field-related values come up with their nil value defaults:
kostas (Tue, 27 Nov 2018 01:42:10 GMT):
`Cluster:{RootCAs:[] ClientCertificate: ClientPrivateKey: DialTimeout:0s RPCTimeout:0s ReplicationBufferSize:0 ReplicationPullTimeout:0s ReplicationRetryTimeout:0s}`
kostas (Tue, 27 Nov 2018 01:51:54 GMT):
Ah, I know what's going on.
kostas (Tue, 27 Nov 2018 01:54:53 GMT):
It's the lack of `Cluster` defaults [here](https://github.com/hyperledger/fabric/blob/69a34181ce4cb8eb890509ec9de3dbb8e77df6e4/orderer/common/localconfig/config.go#L209) that's preventing Steps 1-2-3 above from working as expected.
kostas (Tue, 27 Nov 2018 01:56:08 GMT):
Still, even if `Cluster` can be addressed this way, `Consensus` cannot (because it's an `interface{}`).
kostas (Tue, 27 Nov 2018 01:56:08 GMT):
Still, even if `Cluster` can be addressed this way, `Consensus` cannot, because it's an `interface{}`, so there's really no default value.
kostas (Tue, 27 Nov 2018 01:56:44 GMT):
I think I have a work around.
kostas (Tue, 27 Nov 2018 01:56:44 GMT):
I have a work around.
kostas (Tue, 27 Nov 2018 01:56:55 GMT):
I'm working on this BTW: https://jira.hyperledger.org/browse/FAB-12828
guoger (Tue, 27 Nov 2018 06:18:58 GMT):
as I'm reviewing https://gerrit.hyperledger.org/r/c/27350/25, this edge case occurs to me: tl;dr, if a `raftpb.MsgApp` got lost on wire, (no leadership change), our program hangs.
To expand on this:
- `Order` an env on leader, leader cuts block and invoke `Propose`
- this block data will be replicated to followers via `raftpb.MsgApp`
- `raftpb.MsgApp` got lost
- leader is waiting for this block to come back on `commitC`
etcd/raft handles this by:
- `Propose` is invoked
- it is sent to followers via `MsgApp` (Index: 1)
- this `MsgApp` is lost
- new data comes, `Propose` is invoked again
- it is sent to followers via `MsgApp` (Index: 2)
- follower notices the gap in index (expecting 1, got 2)
- follower replies `MsgAppResp` to reject it, with Index hint set to 1
- leader retransmits Index 1
So, this is not a problem with optimistic block creation, cuz we are not block waiting for previous block to be consented, and next block will trigger the retransmission of previous block. However, we need to solve this with `Config Block`.
If message loss can be detected, maybe we could `Propose` empty data to enforce retransmission.
guoger (Tue, 27 Nov 2018 06:18:58 GMT):
as I'm reviewing https://gerrit.hyperledger.org/r/c/27350/25, this edge case occurs to me: tl;dr, if a `raftpb.MsgApp` got lost on wire, (no leadership change), our program hangs.
To expand on this:
- `Order` an env on leader, leader cuts block and invoke `Propose`
- this block data will be replicated to followers via `raftpb.MsgApp`
- `raftpb.MsgApp` got lost
- leader is waiting for this block to come back on `commitC`
etcd/raft handles this by:
- `Propose` is invoked
- it is sent to followers via `MsgApp` (Index: 1)
- this `MsgApp` is lost
- new data comes, `Propose` is invoked again
- it is sent to followers via `MsgApp` (Index: 2)
- follower notices the gap in index (expecting 1, got 2)
- follower replies `MsgAppResp` to reject it, with Index hint set to 1
- leader retransmits Index 1
So, this is not a problem with optimistic block creation, cuz we are not block waiting for previous block to be consented, and next block will trigger the retransmission of previous block. However, we need to solve this with `Config Block`.
Maybe we could `Propose` empty data to enforce retransmission. But I wanna check if this message loss can be detected in some way?
cc @yacovm @C0rWin @adarshsaraf123
guoger (Tue, 27 Nov 2018 06:18:58 GMT):
as I'm reviewing https://gerrit.hyperledger.org/r/c/27350/25, this edge case occurs to me:
tl;dr, if a `raftpb.MsgApp` got lost on wire, (no leadership change), our program hangs.
To expand on this:
- `Order` an env on leader, leader cuts block and invoke `Propose`
- this block data will be replicated to followers via `raftpb.MsgApp`
- `raftpb.MsgApp` got lost
- leader is waiting for this block to come back on `commitC`
etcd/raft handles this by:
- `Propose` is invoked
- it is sent to followers via `MsgApp` (Index: 1)
- this `MsgApp` is lost
- new data comes, `Propose` is invoked again
- it is sent to followers via `MsgApp` (Index: 2)
- follower notices the gap in index (expecting 1, got 2)
- follower replies `MsgAppResp` to reject it, with Index hint set to 1
- leader retransmits Index 1
So, this is not a problem with optimistic block creation, cuz we are not block waiting for previous block to be consented, and next block will trigger the retransmission of previous block. However, we need to solve this with `Config Block`.
Maybe we could `Propose` empty data to enforce retransmission. But I wanna check if this message loss can be detected in some way?
cc @yacovm @C0rWin @adarshsaraf123
guoger (Tue, 27 Nov 2018 06:18:58 GMT):
as I'm reviewing https://gerrit.hyperledger.org/r/c/27350/25, this edge case occurs to me:
tl;dr, if a `raftpb.MsgApp` got lost on wire, (no leadership change), our program hangs.
To expand on this:
- `Order` an env on leader, leader cuts block and invoke `Propose`
- this block data will be replicated to followers via `raftpb.MsgApp`
- `raftpb.MsgApp` got lost
- leader is infinitely waiting for this block to come back on `commitC`
etcd/raft handles this by:
- `Propose` is invoked
- it is sent to followers via `MsgApp` (Index: 1)
- this `MsgApp` is lost
- new data comes, `Propose` is invoked again
- it is sent to followers via `MsgApp` (Index: 2)
- follower notices the gap in index (expecting 1, got 2)
- follower replies `MsgAppResp` to reject it, with Index hint set to 1
- leader retransmits Index 1
So, this is not a problem with optimistic block creation, cuz we are not block waiting for previous block to be consented, and next block will trigger the retransmission of previous block. However, we need to solve this with `Config Block`.
Maybe we could `Propose` empty data to enforce retransmission. But I wanna check if this message loss can be detected in some way?
cc @yacovm @C0rWin @adarshsaraf123
adarshsaraf123 (Tue, 27 Nov 2018 06:51:51 GMT):
@guoger Can you expand on how `raftpb.MsgApp` can get lost on wire? I would think that should not be possible unless the grpc APIs are not reliable.
guoger (Tue, 27 Nov 2018 06:56:45 GMT):
basically if `rpc.Step` returns error, we probably should consider retransmitting?
C0rWin (Tue, 27 Nov 2018 09:11:04 GMT):
@guoger first of all not sure why this is related to failover, this appears to me as general problem.
guoger (Tue, 27 Nov 2018 09:21:16 GMT):
with optimistic block creation, we _may_ not have this problem with _normal blocks_, but we do need to take care of _config block_, since it goes through two rounds of consensus. But I guess this does not block failover CR still. lift -1 now.
guoger (Tue, 27 Nov 2018 09:21:16 GMT):
with optimistic block creation, we _may_ not have this problem with _normal blocks_, but we do need to take care of _config block_. But I guess this does not block failover CR still. lift -1 now.
kostas (Tue, 27 Nov 2018 14:49:44 GMT):
@guoger: Thanks for posting this.
kostas (Tue, 27 Nov 2018 14:49:54 GMT):
> next block will trigger the retransmission of previous block.
kostas (Tue, 27 Nov 2018 14:50:02 GMT):
Expand on this, I'm not sure I got you here?
kostas (Tue, 27 Nov 2018 14:50:02 GMT):
Expand on this. I'm not sure I got you here?
kostas (Tue, 27 Nov 2018 14:50:02 GMT):
Expand on this? I'm not sure I got you here.
C0rWin (Tue, 27 Nov 2018 16:22:16 GMT):
`rpc.Step` could return success while you still might lose the block, no?
C0rWin (Tue, 27 Nov 2018 16:23:41 GMT):
also, @guoger can you please expand in your scenario who will lose the block? Do you mean the follower will get stack on waiting for block on `commitC`?
C0rWin (Tue, 27 Nov 2018 16:27:54 GMT):
but if `MsgApp` got lost, doesn't that mean you will be waiting for `c.node.Ready()` at first place?
kostas (Tue, 27 Nov 2018 20:47:31 GMT):
Back to disabling/hiding the etcd/raft plugin for 1.4 for a second if I may -
kostas (Tue, 27 Nov 2018 20:47:58 GMT):
At the end of the day, _disabling_ the etcd/raft plugin comes down to remove the "etcdraft" string from this map: https://github.com/hyperledger/fabric/blob/5c8a405dde739567e5e996b8be12db8f1ebecf0a/orderer/common/server/main.go#L67
kostas (Tue, 27 Nov 2018 20:48:43 GMT):
(Hiding includes removing all references from user-facing assets, i.e. config files, and adjusting tests accordingly.)
kostas (Tue, 27 Nov 2018 20:49:16 GMT):
Of course, if I remove that string from `clusterTypes` everything we have out there will break and development will come to a halt.
kostas (Tue, 27 Nov 2018 20:51:11 GMT):
What does everyone thing about pushing two CRs to master:
kostas (Tue, 27 Nov 2018 20:51:38 GMT):
1. CR A removes "etcdraft" from `clusterTypes` -- this is what gets merged to master right away so that we cut 1.4
kostas (Tue, 27 Nov 2018 20:52:38 GMT):
2. CR B is a revert commit of CR A, i.e. re-adds "etcdraft" to `clusterTypes`. We then rebase all pending CRs on top of CR B.
kostas (Tue, 27 Nov 2018 20:52:54 GMT):
Let me know what you think, or if you can think of a less intrusive way of going at it.
kostas (Tue, 27 Nov 2018 20:53:19 GMT):
This is also a reason why I want us to decrease the height of the pending CR stack this week.
kostas (Tue, 27 Nov 2018 21:30:44 GMT):
> In etcdraft, `Step` is a super frequent unary grpc call, and with commit I868f1e13, log is overwhelmed with `step` messages. is it possible to suppress an interceptor for a particular api?
kostas (Tue, 27 Nov 2018 21:30:57 GMT):
@guoger: Matt has merged this: https://gerrit.hyperledger.org/r/c/27700/
C0rWin (Tue, 27 Nov 2018 22:10:06 GMT):
can't we hide raft related staff behind build tags as we did for java shim in early releases?
kostas (Tue, 27 Nov 2018 23:02:16 GMT):
What do you do with Raft-related stuff that coexists with non-Raft related things?
kostas (Tue, 27 Nov 2018 23:02:59 GMT):
For instance: the YAML files, or the integration test file which includes both Raft and non-Raft specs?
kostas (Tue, 27 Nov 2018 23:03:13 GMT):
(I know you can disable individual specs.)
yacovm (Tue, 27 Nov 2018 23:07:29 GMT):
why do we need to hide raft stuff?
yacovm (Tue, 27 Nov 2018 23:07:41 GMT):
it's not like someone can accidentally activate raft
yacovm (Tue, 27 Nov 2018 23:07:44 GMT):
or migrate to it
kostas (Tue, 27 Nov 2018 23:23:37 GMT):
Per David:
kostas (Tue, 27 Nov 2018 23:23:37 GMT):
Per Dave:
kostas (Tue, 27 Nov 2018 23:23:41 GMT):
> We have a policy of not exposing 'experimental' features in official releases anymore, we'll need a way to disable raft from code perspective before we ship v1.4
kostas (Tue, 27 Nov 2018 23:24:21 GMT):
And:
kostas (Tue, 27 Nov 2018 23:24:23 GMT):
> the main requirement is that v1.4 consumers should not be able to enable it. depending on the feature, this often means a few lines of code to disable the entry points.
yacovm (Tue, 27 Nov 2018 23:28:15 GMT):
I say we just add in this function, a panic: https://github.com/hyperledger/fabric/blob/master/orderer/common/server/main.go#L351-L369
yacovm (Tue, 27 Nov 2018 23:28:38 GMT):
this decides whether the OSN is a cluster type
yacovm (Tue, 27 Nov 2018 23:28:45 GMT):
which means it's compatible with etcdraft
yacovm (Tue, 27 Nov 2018 23:29:10 GMT):
so, if you accidentally make a bootstrap genesis block that is of type etcdraft
yacovm (Tue, 27 Nov 2018 23:29:14 GMT):
you'll never going live
yacovm (Tue, 27 Nov 2018 23:29:30 GMT):
you crash
kostas (Tue, 27 Nov 2018 23:30:09 GMT):
That's probably even better than editing `clusterTypes`. Still, we'd be looking at two commits as I describe above, correct?
yacovm (Tue, 27 Nov 2018 23:32:10 GMT):
why 2 commits?
kostas (Tue, 27 Nov 2018 23:33:52 GMT):
One adds the panic (and is the one we merge for 1.4) and one reverts it, and is the one on top of which we add any raft-related code that depends on the `isClusterType` function to work correctly.
kostas (Tue, 27 Nov 2018 23:34:04 GMT):
(That second one doesn't get merged until we cut 1.4)
kostas (Tue, 27 Nov 2018 23:34:04 GMT):
(That second one doesn't get merged until we cut 1.4.)
yacovm (Tue, 27 Nov 2018 23:38:20 GMT):
ah yeah
yacovm (Tue, 27 Nov 2018 23:38:36 GMT):
I can do it now....
yacovm (Tue, 27 Nov 2018 23:38:39 GMT):
@kostas
yacovm (Tue, 27 Nov 2018 23:38:42 GMT):
how bout it?
kostas (Tue, 27 Nov 2018 23:39:46 GMT):
By all means, be my guest. I think we'll need a bit more than that though.
kostas (Tue, 27 Nov 2018 23:40:04 GMT):
So, we need to remove all references to etcd/raft from the YAML files (configtx, orderer).
kostas (Tue, 27 Nov 2018 23:40:17 GMT):
If we do that, `TestSpawnEtcdRaft` will fail.
kostas (Tue, 27 Nov 2018 23:40:43 GMT):
So we need to copy over `orderer.yaml` (w/ some modifications) to `orderer/common/server/testdata`.
kostas (Tue, 27 Nov 2018 23:41:05 GMT):
And we need to disable the E2E integration specs that are related to Raft.
kostas (Tue, 27 Nov 2018 23:41:19 GMT):
At least that's what I've done so far. (I keep running tests.)
kostas (Tue, 27 Nov 2018 23:41:27 GMT):
Am I missing anything?
yacovm (Tue, 27 Nov 2018 23:43:27 GMT):
disabling tests can be done in the same change set
yacovm (Tue, 27 Nov 2018 23:44:08 GMT):
ah is that what you meant?
kostas (Tue, 27 Nov 2018 23:44:17 GMT):
I didn't suggest otherwise. I'm just saying, just doing the panic will fail.
yacovm (Tue, 27 Nov 2018 23:45:38 GMT):
so IMO *not reverting YAMLs* does not contradicts @dave.enyeart 's criteria
yacovm (Tue, 27 Nov 2018 23:45:38 GMT):
so IMO *not reverting YAMLs* does not contradict @dave.enyeart 's criteria
yacovm (Tue, 27 Nov 2018 23:46:01 GMT):
that is - we can just disable etcdraft and keep the YAMLs
yacovm (Tue, 27 Nov 2018 23:46:16 GMT):
@dave.enyeart ?
kostas (Tue, 27 Nov 2018 23:47:12 GMT):
I would think we'll need to hide references from the YAMLs otherwise folks will be trying to activate raft just because they see it as an option. (As you know, nobody RTFM, much less the release notes.)
kostas (Tue, 27 Nov 2018 23:47:16 GMT):
But let's see what Dave says.
yacovm (Tue, 27 Nov 2018 23:49:07 GMT):
if someone specifies a profile in configtx.yaml that activates etcdraft, then the OSN will not boot
yacovm (Tue, 27 Nov 2018 23:49:22 GMT):
I am just worried about merge conflicts, honestly
yacovm (Tue, 27 Nov 2018 23:49:32 GMT):
I don't want to merge the disable CR soon
yacovm (Tue, 27 Nov 2018 23:49:43 GMT):
only before the release...
kostas (Tue, 27 Nov 2018 23:50:30 GMT):
Yeah, I'm aware, same here. Was planning to -2 and let Dave know about the timing anyway.
kostas (Tue, 27 Nov 2018 23:51:05 GMT):
> if someone specifies a profile in configtx.yaml that activates etcdraft, then the OSN will not boot
That someone will be wondering why on earth "raft" is listed as an option?
yacovm (Tue, 27 Nov 2018 23:56:37 GMT):
hmmm but we didn't add it to the `fabric-samples`
yacovm (Tue, 27 Nov 2018 23:56:47 GMT):
(classic excuse, I know)
kostas (Tue, 27 Nov 2018 23:58:40 GMT):
Heh, let's see what Dave says.
kostas (Tue, 27 Nov 2018 23:58:59 GMT):
> If we do that, `TestSpawnEtcdRaft` will fail.
> So we need to copy over `orderer.yaml` (w/ some modifications) to `orderer/common/server/testdata`.
kostas (Tue, 27 Nov 2018 23:59:04 GMT):
I'm obviously wrong here BTW.
kostas (Tue, 27 Nov 2018 23:59:31 GMT):
Because of course that test will fail when `isClusterType` panics, whether the `orderer.yaml` has been copied over or not.
kostas (Wed, 28 Nov 2018 00:02:38 GMT):
So I'll hide https://github.com/hyperledger/fabric/blob/5c8a405dde739567e5e996b8be12db8f1ebecf0a/orderer/common/server/etcdraft_test.go behind a build tag.
yacovm (Wed, 28 Nov 2018 00:03:07 GMT):
we can just disable the test
kostas (Wed, 28 Nov 2018 00:14:44 GMT):
`t.Skip()`?
yacovm (Wed, 28 Nov 2018 01:00:28 GMT):
yes
guoger (Wed, 28 Nov 2018 02:14:58 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=6khZ2K4tWZwCbwM85) @kostas let's say we have leader node A, and follower node B. And retransmission can be explained with following steps:
- client `Propose` data to A
- A replicates it by sending `MsgApp1` (Index: 1) to B
- this message got lost
- client `Propose` data to A
- A replicates it by sending `MsgApp2` (Index: 2) to B
- B is expecting index 1, therefore rejects it by sending `MsgAppResp` that contains `{reject: true, rejectHint: 1}`
- A realizes expected index 1 from rejectHint, and resend `MsgApp1` to B
kostas (Wed, 28 Nov 2018 02:17:20 GMT):
@guoger: I get you. Now walk me over how this is a problem with `Config` blocks? (Or is it Raft configuration transactions?)
guoger (Wed, 28 Nov 2018 02:20:09 GMT):
*without* optimistic block creation, this problem applies to both normal block and config block. Because the 4th step in above scenario never goes through, as `serverReq` is waiting for the first block to be consented before accepting new envelopes.
guoger (Wed, 28 Nov 2018 02:21:33 GMT):
*with* optimistic block creation, proposal of subsequent block triggers retransmission of previously lost block
guoger (Wed, 28 Nov 2018 02:21:33 GMT):
*with* optimistic block creation, proposal of subsequent normal block triggers retransmission of previously lost block
guoger (Wed, 28 Nov 2018 02:22:26 GMT):
however, for config block, we still block waiting for it before processing subsequent envelopes
kostas (Wed, 28 Nov 2018 02:33:40 GMT):
Link to code if you have it handy? No rush. And sorry I'm slow. Been a long day.
kostas (Wed, 28 Nov 2018 02:33:40 GMT):
Link to code snipper that shows the problem in action, if you have it handy? No rush. And sorry I'm slow - been a long day.
guoger (Wed, 28 Nov 2018 02:35:33 GMT):
sure gimme a sec
kostas (Wed, 28 Nov 2018 02:37:22 GMT):
(I'll see this tomorrow as I'm signing off for the day.)
guoger (Wed, 28 Nov 2018 02:37:56 GMT):
as merged code currently stands, when a block is cut, it's [proposed to raft](https://github.com/hyperledger/fabric/blob/5c8a405dde739567e5e996b8be12db8f1ebecf0a/orderer/consensus/etcdraft/chain.go#L536), then block waiting on `commitC` [here](https://github.com/hyperledger/fabric/blob/5c8a405dde739567e5e996b8be12db8f1ebecf0a/orderer/consensus/etcdraft/chain.go#L541), therefore not consuming `submitC` anymore, since it's in the same go routine.
guoger (Wed, 28 Nov 2018 02:38:02 GMT):
sure, gn
guoger (Wed, 28 Nov 2018 02:38:04 GMT):
ttyl
guoger (Wed, 28 Nov 2018 02:38:32 GMT):
i'll have a discussion with @C0rWin when he wakes up
dave.enyeart (Wed, 28 Nov 2018 03:23:30 GMT):
@kostas @yacovm As long as it's disabled, I'm fine if the yaml and test files remain, although it would be helpful to add a comment in there indicating that it is for future use and can not be enabled in v1.4, so as to not confuse people.
yacovm (Wed, 28 Nov 2018 08:45:33 GMT):
@kostas ^
yacovm (Wed, 28 Nov 2018 08:45:40 GMT):
the release manager has spoken
yacovm (Wed, 28 Nov 2018 08:45:48 GMT):
the YAML files can stay :)
huikang (Wed, 28 Nov 2018 15:14:35 GMT):
Hi, I have a quick question about the orderer. When there are multiple channels on the same set of orderers, will the orderer use transactions from different channel to form a single block? Thanks.
yacovm (Wed, 28 Nov 2018 15:15:19 GMT):
no
huikang (Wed, 28 Nov 2018 15:15:40 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=Tr6wne5bKvmkyDuSg) @yacovm ok, thanks
kostas (Wed, 28 Nov 2018 15:20:42 GMT):
> the YAML files can stay
@yacovm: Roger. Feel free to go for it then.
kostas (Wed, 28 Nov 2018 15:20:42 GMT):
> the YAML files can stay
@yacovm: Roger. Feel free to go for it then.
kostas (Wed, 28 Nov 2018 15:22:26 GMT):
> as merged code currently stands, when a block is cut, it's proposed to raft, then block waiting on `commitC` here, therefore not consuming `submitC` anymore, since it's in the same go routine.
kostas (Wed, 28 Nov 2018 15:22:48 GMT):
@C0rWin: What is the conclusion with your discussion with @guoger here?
yacovm (Wed, 28 Nov 2018 15:24:22 GMT):
@kostas I'll do it next week
yacovm (Wed, 28 Nov 2018 15:24:32 GMT):
overloaded with 1.4 stuff right now
C0rWin (Wed, 28 Nov 2018 16:03:14 GMT):
@kostas no conclusions as @guoger was offline most of the day
kostas (Wed, 28 Nov 2018 16:04:42 GMT):
What is your take?
C0rWin (Wed, 28 Nov 2018 16:07:54 GMT):
I need to understand the whole picture thus need to speak with Jay before giving any take on this
C0rWin (Wed, 28 Nov 2018 16:08:43 GMT):
Basically it would be great to have a test which will outline the issue
kostas (Wed, 28 Nov 2018 16:10:02 GMT):
Alright. Just a reminder that we need to close the reconfiguration work (as in have it merged and covering all cases) by the end of this week.
C0rWin (Wed, 28 Nov 2018 16:13:45 GMT):
This issue has nothing to do with reconfiguration and last item has been pushed to gerrit
C0rWin (Wed, 28 Nov 2018 16:19:43 GMT):
Here is the CR which handles restarts during config type B processing: https://gerrit.hyperledger.org/r/#/c/27773/
C0rWin (Wed, 28 Nov 2018 16:19:59 GMT):
@kostas ^^^
haggis (Thu, 29 Nov 2018 08:08:27 GMT):
Has joined the channel.
guoger (Thu, 29 Nov 2018 11:09:52 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=jRntaAqhZgEoJR8jQ) @kostas OK, as I dig a bit deeper, this appears to be a false alert :joy: every time leader receives a `MsgHeartbeatResp`, it retransmits uncommitted entries, if any, to the follower. I've pushed a draft UT to assert this: https://gerrit.hyperledger.org/r/c/27804/ I'll add it to top of CR stack if someone can kindly confirms it
guoger (Thu, 29 Nov 2018 11:10:27 GMT):
sorry to stir the conversation without a thorough investigation :(
guoger (Thu, 29 Nov 2018 11:10:40 GMT):
cc @C0rWin
C0rWin (Thu, 29 Nov 2018 11:18:38 GMT):
@guoger it's better to be alerted and disproof than thinking we are safe w/o attesting the assertions :)
C0rWin (Thu, 29 Nov 2018 11:19:16 GMT):
IMO, I still think we might need to consider introducing timeouts while proposing blocks or conf changes
adarshsaraf123 (Thu, 29 Nov 2018 11:21:24 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=WfcLAttCA3qa5jiaQ) @C0rWin Agreed!!
guoger (Thu, 29 Nov 2018 11:28:11 GMT):
Yup, but that’s to address another problem space and we are safe with current cr stack at least.
C0rWin (Thu, 29 Nov 2018 14:20:17 GMT):
@kostas https://gerrit.hyperledger.org/r/#/c/27773/ added UT to cover the restart (loose of quorum) case + addressed your comments, please have a look
huikang (Thu, 29 Nov 2018 14:53:50 GMT):
Hi, could anyone help to look at my question (https://chat.hyperledger.org/channel/fabric-peer-endorser-committer?msg=fRv7hqHnoN7tC4Dyo) Thanks.
StefanKosc (Fri, 30 Nov 2018 08:55:05 GMT):
Has joined the channel.
StefanKosc (Fri, 30 Nov 2018 08:57:33 GMT):
Hi, I have a general question about network design. Let's assume that I have one channel with two organisations in it. Is it possible that each of organisations hosts its own orderer?
tock (Fri, 30 Nov 2018 11:03:43 GMT):
@kostas please take a look: https://gerrit.hyperledger.org/r/#/c/27770/
guoger (Fri, 30 Nov 2018 13:06:35 GMT):
@StefanKosc yes, although it doesn't make much sense in the context of cft
StefanKosc (Fri, 30 Nov 2018 13:23:53 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=ZAF75fT9SSbaM8Y7q) @guoger thanks for response, what do you mean by cft context?
guoger (Fri, 30 Nov 2018 13:51:54 GMT):
kafka/raft is crash fault tolerant (cft), so having each of orgs running an orderer node only adds operation complexity, but not gaining anything
jyellick (Fri, 30 Nov 2018 21:46:59 GMT):
Looks like there is a Raft related test flake if anyone has a minute to look https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/6664/console
yacovm (Fri, 30 Nov 2018 21:58:45 GMT):
@jyellick I have a fix but it's stuck in a chain of CRs
yacovm (Fri, 30 Nov 2018 21:59:28 GMT):
https://gerrit.hyperledger.org/r/#/c/27602/
yacovm (Fri, 30 Nov 2018 22:00:44 GMT):
I can move it on top of master if it's important enough, but - the failure is rare
kostas (Fri, 30 Nov 2018 22:05:01 GMT):
Or we can merge the stack...
kostas (Mon, 03 Dec 2018 02:42:24 GMT):
Heads up: I will miss tomorrow’s scrum. If I’m not mistaken, the Zoom URL should still work even w/ the host out. @adarshsaraf123 @C0rWin @guoger @tock @yacovm
adarshsaraf123 (Tue, 04 Dec 2018 03:29:42 GMT):
@C0rWin @guoger @yacovm Our CR stack needs some rebasing to make them all align with the latest patch sets from previous CRs
kostas (Tue, 04 Dec 2018 17:15:39 GMT):
@guoger @adarshsaraf123: Thank you for the reviews! _Whenever_ you have a chance or spare cycles, have a look at this one as well and LMK what you think? https://gerrit.hyperledger.org/r/c/27773
adarshsaraf123 (Tue, 04 Dec 2018 17:16:34 GMT):
Sure @kostas That's the next item on my list.
arjitkhullar (Wed, 05 Dec 2018 00:04:00 GMT):
Has joined the channel.
kostas (Wed, 05 Dec 2018 23:22:11 GMT):
A few of us are involved in a 8-9am EST call tomorrow that we can't get out of (for reference, our scrum is at 8:30am EST). I asked to have that 8-9am meeting rescheduled w/o luck. As such, I will have to reschedule tomorrow's scrum - sorry about that. We'll do it on Friday same time if that works for everyone?
C0rWin (Thu, 06 Dec 2018 05:34:57 GMT):
Friday is weekend in Israel
yacovm (Thu, 06 Dec 2018 06:41:59 GMT):
I probably won't attend a Friday scrum either because I don't have anything interesting to talk about. Busy with v1.4 items this week
guoger (Thu, 06 Dec 2018 07:42:58 GMT):
works for me
guoger (Thu, 06 Dec 2018 07:43:57 GMT):
just to make sure i understand this correctly, to write an integration test for etcdraft now, i'll have to locally revert the commit that disables etcdraft?
guoger (Thu, 06 Dec 2018 07:43:57 GMT):
just to make sure i understand this correctly, to write an integration test for etcdraft now, i'll have to locally revert the commit that disables etcdraft? cc @yacovm
yacovm (Thu, 06 Dec 2018 07:45:25 GMT):
@guoger no, I have a commit that reverts it
yacovm (Thu, 06 Dec 2018 07:45:28 GMT):
you need to rebase on top
yacovm (Thu, 06 Dec 2018 07:45:46 GMT):
https://gerrit.hyperledger.org/r/#/c/27909/
guoger (Thu, 06 Dec 2018 07:46:17 GMT):
yep, just found the link. Thanks!
adarshsaraf123 (Thu, 06 Dec 2018 08:10:19 GMT):
I too cannot make it on Friday.
kostas (Thu, 06 Dec 2018 13:37:14 GMT):
No worries, Friday's off then.
kostas (Thu, 06 Dec 2018 13:37:53 GMT):
Please post anything that's blocking you, or you need to draw attention to here.
kostas (Thu, 06 Dec 2018 13:39:18 GMT):
Other than the existing in-progress/in-review (and Kafka migration that @tock's working on), I _think_ this is a list of items we need to have ready within the next couple of weeks: https://jira.hyperledger.org/issues/?jql=labels%20%3D%20raft-must
kostas (Thu, 06 Dec 2018 13:39:50 GMT):
Please have a look and LMK if you think we need to add/remove something to it.
kostas (Mon, 10 Dec 2018 14:10:58 GMT):
https://gerrit.hyperledger.org/r/c/27656/9
kostas (Mon, 10 Dec 2018 14:10:58 GMT):
https://gerrit.hyperledger.org/r/c/27656/
kostas (Mon, 10 Dec 2018 14:11:03 GMT):
https://gerrit.hyperledger.org/r/c/27719/
kostas (Mon, 10 Dec 2018 14:11:15 GMT):
https://gerrit.hyperledger.org/r/c/27773/
kostas (Mon, 10 Dec 2018 14:11:37 GMT):
^^ Let's please review and merge.
guoger (Tue, 11 Dec 2018 17:09:28 GMT):
hey, just arrived at hotel. let me know when you are available to talk :)
guoger (Wed, 12 Dec 2018 16:50:39 GMT):
https://gerrit.hyperledger.org/r/c/28119/ cc @kostas @C0rWin there's a typo in UT and we got lucky that test passes.
C0rWin (Wed, 12 Dec 2018 18:08:26 GMT):
@guoger thanks :thumbsup:
kostas (Wed, 12 Dec 2018 19:46:16 GMT):
This should be an easy one to review: https://gerrit.hyperledger.org/r/c/27804/
tock (Thu, 13 Dec 2018 07:03:14 GMT):
@kostas @yacovm Since 1.4 was cut, let us revive this one and continue the CR: https://gerrit.hyperledger.org/r/#/c/27822/
tock (Thu, 13 Dec 2018 07:03:14 GMT):
@kostas @yacovm @C0rWin Since 1.4 was cut, let us revive this one and continue the CR: https://gerrit.hyperledger.org/r/#/c/27822/
tock (Thu, 13 Dec 2018 07:04:37 GMT):
There is a monster waiting for you after this one, see: https://gerrit.hyperledger.org/r/#/c/28113/
tock (Thu, 13 Dec 2018 07:08:44 GMT):
Also @yacovm - please be kind enough to escort this one all the way home, and merge it to fabric-samples: https://gerrit.hyperledger.org/r/#/c/27770/
tock (Thu, 13 Dec 2018 07:08:44 GMT):
Also @yacovm & @kostas - please be kind enough to escort this one all the way home, and merge it to fabric-samples: https://gerrit.hyperledger.org/r/#/c/27770/
yacovm (Thu, 13 Dec 2018 07:28:11 GMT):
@tock but i +2ed....
tock (Thu, 13 Dec 2018 12:04:06 GMT):
Following Yacov's advice I am going to divide the "monster" into pieces that are easier to digest... see the last patch: https://gerrit.hyperledger.org/r/#/c/28113/ which now only handles the protobuffs.
tock (Thu, 13 Dec 2018 12:04:06 GMT):
Following @yacovm 's advice I am going to divide the "monster" into pieces that are easier to digest... see the last patch: https://gerrit.hyperledger.org/r/#/c/28113/ which now only handles the protobuffs.
yacovm (Thu, 13 Dec 2018 12:23:08 GMT):
Divine and conquer is the way to go
yacovm (Thu, 13 Dec 2018 12:23:08 GMT):
Divide and conquer is the way to go
yacovm (Thu, 13 Dec 2018 13:06:42 GMT):
and i also have a change set of my own here- https://gerrit.hyperledger.org/r/#/c/28107/ , which adds a separate listener for intra-cluster communication.
kostas (Thu, 13 Dec 2018 13:16:53 GMT):
> Following @yacovm 's advice I am going to divide the "monster" into pieces that are easier to digest... see the last patch: https://gerrit.hyperledger.org/r/#/c/28113/ which now only handles the protobuffs.
Right, I saw the multi-K line CR and almost had a heart-attack. Thanks for splitting!
tock (Thu, 13 Dec 2018 19:15:04 GMT):
@aso can you please revisit this one? https://gerrit.hyperledger.org/r/#/c/27822/
aso (Thu, 13 Dec 2018 19:15:04 GMT):
Has joined the channel.
aso (Thu, 13 Dec 2018 19:36:31 GMT):
@tock : I'll wait for @jyellick's comment to be addressed
kostas (Thu, 13 Dec 2018 21:44:02 GMT):
@tock: I wouldn't view that as a blocker. We'll probably move to merge the whole stack only when the corresponding tests are there anyway, so don't let this block you. (You're doing well to ask for reviews though.)
kostas (Thu, 13 Dec 2018 21:44:42 GMT):
Speaking of, not sure if this one's ready for review https://gerrit.hyperledger.org/r/c/28155/1 but if it is, add everyone here as a reviewer so we can have a look. Thanks.
kostas (Thu, 13 Dec 2018 21:44:42 GMT):
Speaking of, not sure if this one's ready for review https://gerrit.hyperledger.org/r/c/28155/1 but if it is, add everyone here as a reviewer so that we know it's good to go - thanks!
tock (Fri, 14 Dec 2018 07:06:22 GMT):
@kostas @jyellick @aso w.r.t. https://gerrit.hyperledger.org/r/#/c/28107/ , I address your comments in full, please take a look.
tock (Fri, 14 Dec 2018 07:06:22 GMT):
@kostas @jyellick @aso w.r.t. https://gerrit.hyperledger.org/r/#/c/28107/ , I addressed your comments in full, please take a look at the latest patch.
amitr (Sat, 15 Dec 2018 02:52:46 GMT):
Has joined the channel.
yacovm (Mon, 17 Dec 2018 10:26:47 GMT):
https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/7191/console
yacovm (Mon, 17 Dec 2018 10:26:51 GMT):
fresh CI failure from today
guoger (Mon, 17 Dec 2018 12:50:09 GMT):
thx, i'll look into it
magar36 (Tue, 18 Dec 2018 22:51:14 GMT):
Posted the same question in fabric-orderer channel but not sure if it is very active:
magar36 (Tue, 18 Dec 2018 22:51:16 GMT):
How is deliver api able to utilize grpc to create a two-way client-server model? We know grpc supports bidirectional communication but typically it's a client-server setup wherein client generates requests and server responds to those requests and all this can happen in parallel on a single tcp connection. In fabric though, orderer dispatches the blocks to the peers which are not a direct consequence of the request coming from the peer.
So peer establishes a grpc connection with the orderer (and invokes the deliver api), sends some data, gets the response - this is typical grpc but after some time orderer is able to use the same connection to send the block to the peer as and when a new block is generated. How does orderer know when to send the block to the peer? It must be keeping the connection details with all the peers in memory somehwere to be able to use those when dispatching the new blocks, right?
magar36 (Tue, 18 Dec 2018 22:56:49 GMT):
Has left the channel.
sanket1211 (Thu, 20 Dec 2018 06:25:28 GMT):
Has joined the channel.
guoger (Fri, 21 Dec 2018 03:17:35 GMT):
@yacovm to echo our scrum discussion yesterday, ginkgo parallelize tests by spawning multiple process, so they don't share vars, see http://onsi.github.io/ginkgo/#parallel-specs
guoger (Fri, 21 Dec 2018 03:17:55 GMT):
(i believe in the same process, tests are still ran in serial)
guoger (Fri, 21 Dec 2018 03:19:58 GMT):
btw, is it possible to print debug logs if a test fails? it's pretty hard to reproduce some flaky tests locally, and i wanna see if CI could output debug logs once it hits a failure. (or should i simply turn on log in tests by default?)
guoger (Fri, 21 Dec 2018 03:19:58 GMT):
btw, is it possible to print debug logs if a test fails? it's pretty hard to reproduce some flaky tests locally, and i wanna see if CI could output debug logs once it hits a failure. (or should i simply turn on log in tests by default? it shouldn't pollute CI logs, right?)
guoger (Fri, 21 Dec 2018 03:25:02 GMT):
Also, i need some pro tips on how to reproduce flaky tests locally... i'm currently using `ginkgo -untilItFails`, but it seems not very productive :(
yacovm (Fri, 21 Dec 2018 08:34:46 GMT):
@guoger i see, that's a surprise.
yacovm (Fri, 21 Dec 2018 08:35:08 GMT):
and it is truly devil's work...
yacovm (Fri, 21 Dec 2018 08:35:48 GMT):
how to reproduce flaky tests locally - i usually reproduce them in CI
yacovm (Fri, 21 Dec 2018 08:35:57 GMT):
by having a custom change set that runs the test 1000 times
yacovm (Fri, 21 Dec 2018 08:36:01 GMT):
and only that test
guoger (Fri, 21 Dec 2018 08:37:21 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=Qkmb3iCujp3uPJkJA) @yacovm great tips! how do you do that? CI configuration or simply loop in the code?
guoger (Fri, 21 Dec 2018 08:38:07 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=AC9MaP79LBAXaZrfh) @yacovm i'm curious to hear the reason
yacovm (Fri, 21 Dec 2018 08:40:10 GMT):
look here https://gerrit.hyperledger.org/r/#/c/28258/4/unit-test/run.sh
yacovm (Fri, 21 Dec 2018 08:40:24 GMT):
the reason is - ginkgo was written by the devil
yacovm (Fri, 21 Dec 2018 08:41:21 GMT):
to tempt innocent developers with a framework that looks nice and productive but it is un-usable when problems arise and you need to debug your test
guoger (Fri, 21 Dec 2018 14:44:59 GMT):
is there an integration test for add/remove node?
yacovm (Fri, 21 Dec 2018 15:50:15 GMT):
oh yeah
yacovm (Fri, 21 Dec 2018 15:50:36 GMT):
we merged today a test that changes ALL the cluster ;) @guoger
guoger (Fri, 21 Dec 2018 15:51:25 GMT):
`etcdraft_config_test.go`?
yacovm (Fri, 21 Dec 2018 16:00:39 GMT):
yes
guoger (Fri, 21 Dec 2018 16:04:23 GMT):
is there a list of commands used for gerrit reply? like `Run VerifyBuild`, etc
kostas (Fri, 21 Dec 2018 16:05:03 GMT):
^^ @rameshthoomu - do we have these recorded anywhere?
rameshthoomu (Fri, 21 Dec 2018 16:05:03 GMT):
Has joined the channel.
yacovm (Fri, 21 Dec 2018 16:05:55 GMT):
it's `Run` and then the name of the test...
yacovm (Fri, 21 Dec 2018 16:06:22 GMT):
```
F1-VerifyBuild
F2-DocBuild
F3-IntegrationTest
F3-UnitTest
```
rameshthoomu (Fri, 21 Dec 2018 22:49:12 GMT):
@kostas @guoger Please see this https://ci-docs.readthedocs.io/en/latest/source/fabric_ci_process.html#trigger-failed-jobs-through-gerrit-comments
guoger (Mon, 24 Dec 2018 13:27:35 GMT):
Hi guys, I’m not able to make to our scrum today. I’m taking Mon and Tue off. I put 3 CRs out there for FAB-13178, but -1 the last myself cuz there is a corner case needed to be taken care of. Otherwise, happy holiday!!
adarshsaraf123 (Mon, 24 Dec 2018 13:45:44 GMT):
Happy Holidays!
yacovm (Mon, 24 Dec 2018 17:52:05 GMT):
I think we have a problem... in the `deliver.go`, the `erroredChan` , returned by the `chain.Errored()` method - does *not* close when the cluster loses the leader :(
It means that if a peer is connected to an orderer node and that orderer is in a network partition and cannot connect to the cluster because reasons,
the peer will not try to failover to another OSN in order to check if it can pull blocks from some other OSN, and since the OSN the peer is connected to, is disconnected from the cluster,
then it will not get any blocks in the meantine....
I suggest that this is solved by using some other channel than `doneC` when `Errored()` is invoked.
let's say that we have a channel that is initialized when `Errored()` is called, and then we keep it until we detect that we don't know who is the leader (which, we need to track too... I don't think we currently do that actively).
when we detect that the leader is lost, we need to close the channel we gave to the goroutines that invokes the `Errored()`, and create a new channel on demand.
something roughly like this:
```
func (c *Chain) Errored() <-chan struct{} {
lead := atomic.LoadUint64(&c.leader)
if lead == raft.None {
closedChan := make(chan struct{})
close(closedChan)
return closedChan
}
// Else, we have a leader.
c.syncLock.Lock()
erroredChan := c.erroredChan
if erroredChan == nil {
erroredChan = make(chan struct{})
}
c.syncLock.Unlock()
return erroredChan
}
```
yacovm (Mon, 24 Dec 2018 17:52:21 GMT):
@guoger @kostas @C0rWin @tock @adarshsaraf123
yacovm (Mon, 24 Dec 2018 17:57:42 GMT):
I opened https://jira.hyperledger.org/browse/FAB-13438
yacovm (Mon, 24 Dec 2018 17:57:49 GMT):
In case I am not wrong, that is
yacovm (Mon, 24 Dec 2018 17:58:01 GMT):
To track/discuss this
guoger (Thu, 27 Dec 2018 10:01:20 GMT):
that seems to be a reasonable solution
guoger (Thu, 27 Dec 2018 10:01:32 GMT):
i've linked FAB-12709 to the jira you created to address this:
> and then we keep it until we detect that we don't know who is the leader (which, we need to track too... I don't think we currently do that actively).
guoger (Thu, 27 Dec 2018 10:02:02 GMT):
i'll do that as part of my WIP stack
yacovm (Fri, 28 Dec 2018 21:11:59 GMT):
Something is off with the chain, I think.... @guoger
https://gerrit.hyperledger.org/r/#/c/28391/8/integration/e2e/etcdraft_reconfig_test.go
In this test - I bring up a new OSN - orderer4, and I send a transaction through it:
```
By("Submitting a transaction through orderer4")
assertInvoke(network, peer, o4, mycc2.Name, "testchannel2", "Chaincode invoke successful. result: status:200", 0)
By("And ensuring it is propagated amongst all orderers")
assertBlockReception(map[string]int{
"testchannel2": 3,
}, orderers, peer, network)
```
All orderers but that OSN commit the transaction :/
```
[e][OrdererOrg.orderer4] 2018-12-28 22:55:26.751 IST [orderer.consensus.etcdraft] run -> INFO 1ee raft.node: 4 elected leader 1 at term 2 {"channel": "testchannel2", "node": 4}
[e][OrdererOrg.orderer4] 2018-12-28 22:55:26.751 IST [orderer.consensus.etcdraft] serveRaft -> INFO 1ef Raft leader changed: 0 -> 1 {"channel": "testchannel2", "node": 4}
[e][OrdererOrg.orderer4] 2018-12-28 22:55:26.761 IST [orderer.consensus.etcdraft] Step -> INFO 1f0 7 [term: 1] received a MsgHeartbeat message with higher term from 4 [term: 3] {"channel": "testchannel", "
node": 7}
[e][OrdererOrg.orderer4] 2018-12-28 22:55:26.761 IST [orderer.consensus.etcdraft] becomeFollower -> INFO 1f1 7 became follower at term 3 {"channel": "testchannel", "node": 7}
[e][OrdererOrg.orderer4] 2018-12-28 22:55:26.761 IST [orderer.consensus.etcdraft] run -> INFO 1f2 raft.node: 7 elected leader 4 at term 3 {"channel": "testchannel", "node": 7}
[e][OrdererOrg.orderer4] 2018-12-28 22:55:26.762 IST [orderer.consensus.etcdraft] Step -> INFO 1f3 7 [term: 1] received a MsgHeartbeat message with higher term from 4 [term: 4] {"channel": "systemchannel",
"node": 7}
[e][OrdererOrg.orderer4] 2018-12-28 22:55:26.762 IST [orderer.consensus.etcdraft] becomeFollower -> INFO 1f4 7 became follower at term 4 {"channel": "systemchannel", "node": 7}
[e][OrdererOrg.orderer4] 2018-12-28 22:55:26.762 IST [orderer.consensus.etcdraft] run -> INFO 1f5 raft.node: 7 elected leader 4 at term 4 {"channel": "systemchannel", "node": 7}
[e][OrdererOrg.orderer4] 2018-12-28 22:55:26.766 IST [orderer.consensus.etcdraft] serveRaft -> INFO 1f6 Raft leader changed: 0 -> 4 {"channel": "testchannel", "node": 7}
[e][OrdererOrg.orderer4] 2018-12-28 22:55:26.767 IST [orderer.consensus.etcdraft] serveRaft -> INFO 1f7 Raft leader changed: 0 -> 4 {"channel": "systemchannel", "node": 7}
STEP: Submitting a transaction through orderer4
[e][Org1.peer1] 2018-12-28 22:55:36.120 IST [endorser] callChaincode -> INFO 0e0 [testchannel2][b504068d] Entry chaincode: name:"mycc2"
[e][Org1.peer1] 2018-12-28 22:55:36.121 IST [peer.chaincode.7c26ujzkrnf45ha2nnbb2y4o6m-Org1.peer1-mycc2-0.0] func2 -> INFO 0e1 ex02 Invoke
[e][Org1.peer1] 2018-12-28 22:55:36.122 IST [peer.chaincode.7c26ujzkrnf45ha2nnbb2y4o6m-Org1.peer1-mycc2-0.0] func2 -> INFO 0e2 Aval = 90, Bval = 210
[e][Org1.peer1] 2018-12-28 22:55:36.122 IST [endorser] callChaincode -> INFO 0e3 [testchannel2][b504068d] Exit chaincode: name:"mycc2" (2ms)
[e][Org1.peer1] 2018-12-28 22:55:36.122 IST [comm.grpc.server] 1 -> INFO 0e4 unary call completed {"grpc.start_time": "2018-12-28T22:55:36.119+02:00", "grpc.service": "protos.Endorser", "grpc.method": "Pro
cessProposal", "grpc.peer_address": "127.0.0.1:47110", "grpc.code": "OK", "grpc.call_duration": "3.543639ms"}
[e][Org1.peer1] 2018-12-28 22:55:36.126 IST [common.deliver] deliverBlocks -> INFO 0e5 [channel: testchannel2] Delivering block for (0xc00bd22a80) for 127.0.0.1:47111
[o][OrdererOrg.orderer1] <<<<<< c.commitC <- block{ 3 }
[o][OrdererOrg.orderer1] <<<<<< writeBlock( 3 )
[o][OrdererOrg.orderer1] >>>>>>>>>>> writing block 3 for channel testchannel2
[o][OrdererOrg.orderer2] <<<<<< c.commitC <- block{ 3 }
[o][OrdererOrg.orderer2] <<<<<< writeBlock( 3 )
[o][OrdererOrg.orderer2] >>>>>>>>>>> writing block 3 for channel testchannel2
[o][OrdererOrg.orderer3] <<<<<< c.commitC <- block{ 3 }
[o][OrdererOrg.orderer3] <<<<<< writeBlock( 3 )
[o][OrdererOrg.orderer3] >>>>>>>>>>> writing block 3 for channel testchannel2
```
yacovm (Fri, 28 Dec 2018 21:11:59 GMT):
Something is off with the chain, I think.... @guoger
https://gerrit.hyperledger.org/r/#/c/28391/8/integration/e2e/etcdraft_reconfig_test.go
In this test - I bring up a new OSN - orderer4, and I send a transaction through it:
```
By("Submitting a transaction through orderer4")
assertInvoke(network, peer, o4, mycc2.Name, "testchannel2", "Chaincode invoke successful. result: status:200", 0)
By("And ensuring it is propagated amongst all orderers")
assertBlockReception(map[string]int{
"testchannel2": 3,
}, orderers, peer, network)
```
All orderers but that OSN commit the block :/
```
[e][OrdererOrg.orderer4] 2018-12-28 22:55:26.751 IST [orderer.consensus.etcdraft] run -> INFO 1ee raft.node: 4 elected leader 1 at term 2 {"channel": "testchannel2", "node": 4}
[e][OrdererOrg.orderer4] 2018-12-28 22:55:26.751 IST [orderer.consensus.etcdraft] serveRaft -> INFO 1ef Raft leader changed: 0 -> 1 {"channel": "testchannel2", "node": 4}
[e][OrdererOrg.orderer4] 2018-12-28 22:55:26.761 IST [orderer.consensus.etcdraft] Step -> INFO 1f0 7 [term: 1] received a MsgHeartbeat message with higher term from 4 [term: 3] {"channel": "testchannel", "
node": 7}
[e][OrdererOrg.orderer4] 2018-12-28 22:55:26.761 IST [orderer.consensus.etcdraft] becomeFollower -> INFO 1f1 7 became follower at term 3 {"channel": "testchannel", "node": 7}
[e][OrdererOrg.orderer4] 2018-12-28 22:55:26.761 IST [orderer.consensus.etcdraft] run -> INFO 1f2 raft.node: 7 elected leader 4 at term 3 {"channel": "testchannel", "node": 7}
[e][OrdererOrg.orderer4] 2018-12-28 22:55:26.762 IST [orderer.consensus.etcdraft] Step -> INFO 1f3 7 [term: 1] received a MsgHeartbeat message with higher term from 4 [term: 4] {"channel": "systemchannel",
"node": 7}
[e][OrdererOrg.orderer4] 2018-12-28 22:55:26.762 IST [orderer.consensus.etcdraft] becomeFollower -> INFO 1f4 7 became follower at term 4 {"channel": "systemchannel", "node": 7}
[e][OrdererOrg.orderer4] 2018-12-28 22:55:26.762 IST [orderer.consensus.etcdraft] run -> INFO 1f5 raft.node: 7 elected leader 4 at term 4 {"channel": "systemchannel", "node": 7}
[e][OrdererOrg.orderer4] 2018-12-28 22:55:26.766 IST [orderer.consensus.etcdraft] serveRaft -> INFO 1f6 Raft leader changed: 0 -> 4 {"channel": "testchannel", "node": 7}
[e][OrdererOrg.orderer4] 2018-12-28 22:55:26.767 IST [orderer.consensus.etcdraft] serveRaft -> INFO 1f7 Raft leader changed: 0 -> 4 {"channel": "systemchannel", "node": 7}
STEP: Submitting a transaction through orderer4
[e][Org1.peer1] 2018-12-28 22:55:36.120 IST [endorser] callChaincode -> INFO 0e0 [testchannel2][b504068d] Entry chaincode: name:"mycc2"
[e][Org1.peer1] 2018-12-28 22:55:36.121 IST [peer.chaincode.7c26ujzkrnf45ha2nnbb2y4o6m-Org1.peer1-mycc2-0.0] func2 -> INFO 0e1 ex02 Invoke
[e][Org1.peer1] 2018-12-28 22:55:36.122 IST [peer.chaincode.7c26ujzkrnf45ha2nnbb2y4o6m-Org1.peer1-mycc2-0.0] func2 -> INFO 0e2 Aval = 90, Bval = 210
[e][Org1.peer1] 2018-12-28 22:55:36.122 IST [endorser] callChaincode -> INFO 0e3 [testchannel2][b504068d] Exit chaincode: name:"mycc2" (2ms)
[e][Org1.peer1] 2018-12-28 22:55:36.122 IST [comm.grpc.server] 1 -> INFO 0e4 unary call completed {"grpc.start_time": "2018-12-28T22:55:36.119+02:00", "grpc.service": "protos.Endorser", "grpc.method": "Pro
cessProposal", "grpc.peer_address": "127.0.0.1:47110", "grpc.code": "OK", "grpc.call_duration": "3.543639ms"}
[e][Org1.peer1] 2018-12-28 22:55:36.126 IST [common.deliver] deliverBlocks -> INFO 0e5 [channel: testchannel2] Delivering block for (0xc00bd22a80) for 127.0.0.1:47111
[o][OrdererOrg.orderer1] <<<<<< c.commitC <- block{ 3 }
[o][OrdererOrg.orderer1] <<<<<< writeBlock( 3 )
[o][OrdererOrg.orderer1] >>>>>>>>>>> writing block 3 for channel testchannel2
[o][OrdererOrg.orderer2] <<<<<< c.commitC <- block{ 3 }
[o][OrdererOrg.orderer2] <<<<<< writeBlock( 3 )
[o][OrdererOrg.orderer2] >>>>>>>>>>> writing block 3 for channel testchannel2
[o][OrdererOrg.orderer3] <<<<<< c.commitC <- block{ 3 }
[o][OrdererOrg.orderer3] <<<<<< writeBlock( 3 )
[o][OrdererOrg.orderer3] >>>>>>>>>>> writing block 3 for channel testchannel2
```
yacovm (Fri, 28 Dec 2018 21:12:03 GMT):
Any idea why?
yacovm (Fri, 28 Dec 2018 21:14:51 GMT):
As shown in the logs above - orderer4 clearly participates in `testchannel2` - it knows that `1` is the leader of `testchannel2`, and the transaction is clearly sent through orderer4, because after the block is committed, the peer CLI exists and the log prints:
```
[e][OrdererOrg.orderer4] 2018-12-28 22:55:36.154 IST [orderer.common.broadcast] Handle -> WARN 1f8 Error reading from 127.0.0.1:40790: rpc error: code = Canceled desc = context canceled
[e][OrdererOrg.orderer4] 2018-12-28 22:55:36.154 IST [comm.grpc.server] 1 -> INFO 1f9 streaming call completed {"grpc.start_time": "2018-12-28T22:55:36.119+02:00", "grpc.service": "orderer.AtomicBroadcast"
, "grpc.method": "Broadcast", "grpc.peer_address": "127.0.0.1:40790", "error": "rpc error: code = Canceled desc = context canceled", "grpc.code": "Canceled", "grpc.call_duration": "35.331141ms"}
```
which is an abrupt connection close from the client side, when the server side is orderer4
yacovm (Fri, 28 Dec 2018 21:16:13 GMT):
I added the `<<<<` prints to see if the `commitC` channel gets the block, and it seems that in `orderer4` it doesn't.
yacovm (Fri, 28 Dec 2018 21:16:48 GMT):
Now, given the fact that the transaction was sent through orderer4 I deduce that the cluster communication must be OK.... any idea what's the problem, @guoger ?
guoger (Sat, 29 Dec 2018 06:45:49 GMT):
@yacovm do you know how to change log level to debug on orderer4?
yacovm (Sat, 29 Dec 2018 07:49:27 GMT):
Yeah... Via some env var called logging_spec
guoger (Sat, 29 Dec 2018 08:17:00 GMT):
in nwo?
yacovm (Sat, 29 Dec 2018 08:25:47 GMT):
In fabric in general
yacovm (Sat, 29 Dec 2018 08:25:59 GMT):
Look in the documentation, it is documented
yacovm (Sat, 29 Dec 2018 08:26:18 GMT):
I'm on the phone so can't really link
guoger (Sat, 29 Dec 2018 08:26:55 GMT):
oh, thx. I was looking at nwo and trying to find the log level switch in yamls..
yacovm (Sat, 29 Dec 2018 08:27:07 GMT):
Any idea what can be the problem? :thinking:
guoger (Sat, 29 Dec 2018 08:27:17 GMT):
sill trying to diagnose...
yacovm (Sat, 29 Dec 2018 09:39:51 GMT):
I think something is wrong with the snapshotting mechanism:
```
[e][Org1.peer1] 2018-12-29 11:26:25.202 IST [endorser] callChaincode -> INFO 0dc [testchannel2][4dbf7885] Entry chaincode: name:"mycc2"
[e][Org1.peer1] 2018-12-29 11:26:25.203 IST [peer.chaincode.q2tmhlbjifcsjel3zp2juyohcq-Org1.peer1-mycc2-0.0] func2 -> INFO 0dd ex02 Invoke
[e][Org1.peer1] 2018-12-29 11:26:25.203 IST [peer.chaincode.q2tmhlbjifcsjel3zp2juyohcq-Org1.peer1-mycc2-0.0] func2 -> INFO 0de Aval = 90, Bval = 210
[e][Org1.peer1] 2018-12-29 11:26:25.204 IST [endorser] callChaincode -> INFO 0df [testchannel2][4dbf7885] Exit chaincode: name:"mycc2" (2ms)
[e][Org1.peer1] 2018-12-29 11:26:25.204 IST [comm.grpc.server] 1 -> INFO 0e0 unary call completed {"grpc.start_time": "2018-12-29T11:26:25.201+02:00", "grpc.service": "protos.Endorser", "grpc.method": "Pro
cessProposal", "grpc.peer_address": "127.0.0.1:43522", "grpc.code": "OK", "grpc.call_duration": "3.752881ms"}
[o][OrdererOrg.orderer3] >>>>> sending 2 messages: [to: 2, type: MsgApp to: 1, type: MsgApp]
[o][OrdererOrg.orderer2] >>>>> sending 1 messages: [to: 3, type: MsgAppResp]
[o][OrdererOrg.orderer1] >>>>> sending 1 messages: [to: 3, type: MsgAppResp]
[o][OrdererOrg.orderer3] >>>>> going to apply 1 entries of size 3079
[o][OrdererOrg.orderer3] >>>>> sending 2 messages: [to: 2, type: MsgApp to: 1, type: MsgApp]
[o][OrdererOrg.orderer2] >>>>> going to apply 1 entries of size 3079
[o][OrdererOrg.orderer2] >>>>> sending 1 messages: [to: 3, type: MsgAppResp]
[o][OrdererOrg.orderer1] >>>>> going to apply 1 entries of size 3079
[o][OrdererOrg.orderer1] >>>>> sending 1 messages: [to: 3, type: MsgAppResp]
[e][Org1.peer1] 2018-12-29 11:26:25.221 IST [gossip.privdata] StoreBlock -> INFO 0e1 [testchannel2] Received block [3] from buffer
[e][Org1.peer1] 2018-12-29 11:26:25.222 IST [committer.txvalidator] Validate -> INFO 0e2 [testchannel2] Validated block [3] in 1ms
[o][OrdererOrg.orderer3] >>>>> sending 3 messages: [to: 2, type: MsgHeartbeat to: 1, type: MsgHeartbeat to: 4, type: MsgHeartbeat]
[o][OrdererOrg.orderer2] >>>>> sending 1 messages: [to: 3, type: MsgHeartbeatResp]
[o][OrdererOrg.orderer1] >>>>> sending 1 messages: [to: 3, type: MsgHeartbeatResp]
[o][OrdererOrg.orderer4] >>>>> sending 1 messages: [to: 3, type: MsgHeartbeatResp]
[o][OrdererOrg.orderer3] >>>>> sending 1 messages: [to: 4, type: MsgApp]
[o][OrdererOrg.orderer4] >>>>> sending 1 messages: [to: 3, type: MsgAppResp]
[o][OrdererOrg.orderer3] >>>>> sending 1 messages: [to: 4, type: MsgApp]
[o][OrdererOrg.orderer4] >>>>> sending 1 messages: [to: 3, type: MsgAppResp]
```
yacovm (Sat, 29 Dec 2018 09:39:51 GMT):
```
[e][Org1.peer1] 2018-12-29 11:26:25.202 IST [endorser] callChaincode -> INFO 0dc [testchannel2][4dbf7885] Entry chaincode: name:"mycc2"
[e][Org1.peer1] 2018-12-29 11:26:25.203 IST [peer.chaincode.q2tmhlbjifcsjel3zp2juyohcq-Org1.peer1-mycc2-0.0] func2 -> INFO 0dd ex02 Invoke
[e][Org1.peer1] 2018-12-29 11:26:25.203 IST [peer.chaincode.q2tmhlbjifcsjel3zp2juyohcq-Org1.peer1-mycc2-0.0] func2 -> INFO 0de Aval = 90, Bval = 210
[e][Org1.peer1] 2018-12-29 11:26:25.204 IST [endorser] callChaincode -> INFO 0df [testchannel2][4dbf7885] Exit chaincode: name:"mycc2" (2ms)
[e][Org1.peer1] 2018-12-29 11:26:25.204 IST [comm.grpc.server] 1 -> INFO 0e0 unary call completed {"grpc.start_time": "2018-12-29T11:26:25.201+02:00", "grpc.service": "protos.Endorser", "grpc.method": "Pro
cessProposal", "grpc.peer_address": "127.0.0.1:43522", "grpc.code": "OK", "grpc.call_duration": "3.752881ms"}
[o][OrdererOrg.orderer3] >>>>> sending 2 messages: [to: 2, type: MsgApp to: 1, type: MsgApp]
[o][OrdererOrg.orderer2] >>>>> sending 1 messages: [to: 3, type: MsgAppResp]
[o][OrdererOrg.orderer1] >>>>> sending 1 messages: [to: 3, type: MsgAppResp]
[o][OrdererOrg.orderer3] >>>>> going to apply 1 entries of size 3079
[o][OrdererOrg.orderer3] >>>>> sending 2 messages: [to: 2, type: MsgApp to: 1, type: MsgApp]
[o][OrdererOrg.orderer2] >>>>> going to apply 1 entries of size 3079
[o][OrdererOrg.orderer2] >>>>> sending 1 messages: [to: 3, type: MsgAppResp]
[o][OrdererOrg.orderer1] >>>>> going to apply 1 entries of size 3079
[o][OrdererOrg.orderer1] >>>>> sending 1 messages: [to: 3, type: MsgAppResp]
[e][Org1.peer1] 2018-12-29 11:26:25.221 IST [gossip.privdata] StoreBlock -> INFO 0e1 [testchannel2] Received block [3] from buffer
[e][Org1.peer1] 2018-12-29 11:26:25.222 IST [committer.txvalidator] Validate -> INFO 0e2 [testchannel2] Validated block [3] in 1ms
[o][OrdererOrg.orderer3] >>>>> sending 3 messages: [to: 2, type: MsgHeartbeat to: 1, type: MsgHeartbeat to: 4, type: MsgHeartbeat]
[o][OrdererOrg.orderer2] >>>>> sending 1 messages: [to: 3, type: MsgHeartbeatResp]
[o][OrdererOrg.orderer1] >>>>> sending 1 messages: [to: 3, type: MsgHeartbeatResp]
[o][OrdererOrg.orderer4] >>>>> sending 1 messages: [to: 3, type: MsgHeartbeatResp]
[o][OrdererOrg.orderer3] >>>>> sending 1 messages: [to: 4, type: MsgApp]
[o][OrdererOrg.orderer4] >>>>> sending 1 messages: [to: 3, type: MsgAppResp]
[o][OrdererOrg.orderer3] >>>>> sending 1 messages: [to: 4, type: MsgApp]
[o][OrdererOrg.orderer4] >>>>> sending 1 messages: [to: 3, type: MsgAppResp]
```
guoger (Sat, 29 Dec 2018 09:59:32 GMT):
orderer4 keeps _rejecting_ MsgApp for block 3 from leader in `testchannel2`, however leader does not have any older data to offer... i'll keep digging..
yacovm (Sat, 29 Dec 2018 10:03:05 GMT):
OK, thanks
yacovm (Sat, 29 Dec 2018 12:10:07 GMT):
looks like the leader tries to sync `orderer4` with entries of term 2 but it rejects them, and then it tries to sync with term 1 and it doesn't reject them.
after a while, it sends again, term2, and it rejects again, etc. etc.
```
[o][OrdererOrg.orderer2] testchannel2 >>>>> sending 1 messages: [to: 4, type: MsgApp, term 2, reject? false, size: 38400, entries: [EntryNormal EntryNormal EntryConfChange EntryNormal]]
[o][OrdererOrg.orderer4] testchannel2 >>>>> sending 1 messages: [to: 2, type: MsgAppResp, term 0, reject? true, size: 28, entries: []]
[o][OrdererOrg.orderer2] testchannel2 >>>>> sending 1 messages: [to: 4, type: MsgApp, term 1, reject? false, size: 38408, entries: [EntryNormal EntryNormal EntryNormal EntryConfChange EntryNormal]]
[o][OrdererOrg.orderer4] testchannel2 >>>>> sending 1 messages: [to: 2, type: MsgAppResp, term 0, reject? false, size: 28, entries: []]
```
[o][OrdererOrg.orderer2] testchannel2 >>>>> sending 1 messages: [to: 4, type: MsgApp, term 2, reject? false, size: 38400, entries: [EntryNormal EntryNormal EntryConfChange EntryNormal]]
[o][OrdererOrg.orderer4] testchannel2 >>>>> sending 1 messages: [to: 2, type: MsgAppResp, term 0, reject? true, size: 28, entries: []]
[o][OrdererOrg.orderer2] testchannel2 >>>>> sending 1 messages: [to: 4, type: MsgApp, term 1, reject? false, size: 38408, entries: [EntryNormal EntryNormal EntryNormal EntryConfChange EntryNormal]]
[o][OrdererOrg.orderer4] testchannel2 >>>>> sending 1 messages: [to: 2, type: MsgAppResp, term 0, reject? false, size: 28, entries: []]
```
```
yacovm (Sat, 29 Dec 2018 12:10:07 GMT):
looks like the leader tries to sync `orderer4` with entries of term 2 but it rejects them, and then it tries to sync with term 1 and it doesn't reject them.
after a while, it sends again, term2, and it rejects again, etc. etc.
```
[o][OrdererOrg.orderer2] testchannel2 >>>>> sending 1 messages: [to: 4, type: MsgApp, term 2, reject? false, size: 38400, entries: [EntryNormal EntryNormal EntryConfChange EntryNormal]]
[o][OrdererOrg.orderer4] testchannel2 >>>>> sending 1 messages: [to: 2, type: MsgAppResp, term 0, reject? true, size: 28, entries: []]
[o][OrdererOrg.orderer2] testchannel2 >>>>> sending 1 messages: [to: 4, type: MsgApp, term 1, reject? false, size: 38408, entries: [EntryNormal EntryNormal EntryNormal EntryConfChange EntryNormal]]
[o][OrdererOrg.orderer4] testchannel2 >>>>> sending 1 messages: [to: 2, type: MsgAppResp, term 0, reject? false, size: 28, entries: []]
```
[o][OrdererOrg.orderer2] testchannel2 >>>>> sending 1 messages: [to: 4, type: MsgApp, term 2, reject? false, size: 38400, entries: [EntryNormal EntryNormal EntryConfChange EntryNormal]]
[o][OrdererOrg.orderer4] testchannel2 >>>>> sending 1 messages: [to: 2, type: MsgAppResp, term 0, reject? true, size: 28, entries: []]
[o][OrdererOrg.orderer2] testchannel2 >>>>> sending 1 messages: [to: 4, type: MsgApp, term 1, reject? false, size: 38408, entries: [EntryNormal EntryNormal EntryNormal EntryConfChange EntryNormal]]
[o][OrdererOrg.orderer4] testchannel2 >>>>> sending 1 messages: [to: 2, type: MsgAppResp, term 0, reject? false, size: 28, entries: []]
```
yacovm (Sat, 29 Dec 2018 12:10:07 GMT):
looks like the leader tries to sync `orderer4` with entries of term 2 but it rejects them, and then it tries to sync with term 1 and it doesn't reject them.
after a while, it sends again, term2, and it rejects again, etc. etc.
```
[o][OrdererOrg.orderer2] testchannel2 >>>>> sending 1 messages: [to: 4, type: MsgApp, term 2, reject? false, size: 38400, entries: [EntryNormal EntryNormal EntryConfChange EntryNormal]]
[o][OrdererOrg.orderer4] testchannel2 >>>>> sending 1 messages: [to: 2, type: MsgAppResp, term 0, reject? true, size: 28, entries: []]
[o][OrdererOrg.orderer2] testchannel2 >>>>> sending 1 messages: [to: 4, type: MsgApp, term 1, reject? false, size: 38408, entries: [EntryNormal EntryNormal EntryNormal EntryConfChange EntryNormal]]
[o][OrdererOrg.orderer4] testchannel2 >>>>> sending 1 messages: [to: 2, type: MsgAppResp, term 0, reject? false, size: 28, entries: []]
```
```
[o][OrdererOrg.orderer2] testchannel2 >>>>> sending 1 messages: [to: 4, type: MsgApp, term 2, reject? false, size: 38400, entries: [EntryNormal EntryNormal EntryConfChange EntryNormal]]
[o][OrdererOrg.orderer4] testchannel2 >>>>> sending 1 messages: [to: 2, type: MsgAppResp, term 0, reject? true, size: 28, entries: []]
[o][OrdererOrg.orderer2] testchannel2 >>>>> sending 1 messages: [to: 4, type: MsgApp, term 1, reject? false, size: 38408, entries: [EntryNormal EntryNormal EntryNormal EntryConfChange EntryNormal]]
[o][OrdererOrg.orderer4] testchannel2 >>>>> sending 1 messages: [to: 2, type: MsgAppResp, term 0, reject? false, size: 28, entries: []]
```
yacovm (Sat, 29 Dec 2018 12:10:38 GMT):
@guoger any idea what's going on?
guoger (Sat, 29 Dec 2018 13:53:32 GMT):
problem still remains unsolved, but one thing i could currently see is that this message pattern can be observed with our [type b config UT](https://github.com/hyperledger/fabric/blob/d3e80320e5bd97bf7e1232f327be6377c66e3d5f/orderer/consensus/etcdraft/chain_test.go#L1375). Rejection goes away when a new message is proposed in UT. still trying to figure out why 1) new node keeps rejecting MsgApp 2) why block is not committed in integration tests
guoger (Sat, 29 Dec 2018 13:53:32 GMT):
problem still remains unsolved, but one thing i could currently see is that this message pattern can be observed with our [type b config UT](https://github.com/hyperledger/fabric/blob/d3e80320e5bd97bf7e1232f327be6377c66e3d5f/orderer/consensus/etcdraft/chain_test.go#L1375). Although _Rejection goes away when a new message is proposed in UT_. still trying to figure out why 1) new node keeps rejecting MsgApp 2) why block is not committed in integration tests
guoger (Sat, 29 Dec 2018 13:53:32 GMT):
problem still remains unsolved, but one thing i could currently see is that this message pattern can be observed with our [type b config UT](https://github.com/hyperledger/fabric/blob/d3e80320e5bd97bf7e1232f327be6377c66e3d5f/orderer/consensus/etcdraft/chain_test.go#L1375). ~Although _Rejection goes away when a new message is proposed in UT_~ leader simply probes new follower for its current state. still trying to figure out why ~1) new node keeps rejecting MsgApp~ 2) why block is not committed in integration tests
yacovm (Sat, 29 Dec 2018 15:10:01 GMT):
what do you mean the message pattern can be observed? @guoger ?
yacovm (Sat, 29 Dec 2018 15:10:16 GMT):
that is sends `iterm i` and then term `i-1` ?
guoger (Sat, 29 Dec 2018 15:11:13 GMT):
when new node joins network, leader tries to replicate latest entry to it, which will be rejected. then leader decrease to lower term/index, and try again
yacovm (Sat, 29 Dec 2018 15:11:32 GMT):
I see
guoger (Sat, 29 Dec 2018 16:42:38 GMT):
so, the problem is, when joining a node to existing network, raft should be booted with empty `startPeers`, see https://github.com/etcd-io/etcd/blob/cc8d446a6ec334d7a10a291279b77389b4f2faf7/contrib/raftexample/raft.go#L290-L292
guoger (Sat, 29 Dec 2018 16:42:38 GMT):
so, the problem is, when joining a node to existing network, raft should be booted with empty `startPeers`, see [this](https://github.com/etcd-io/etcd/blob/cc8d446a6ec334d7a10a291279b77389b4f2faf7/contrib/raftexample/raft.go#L290-L292)
guoger (Sat, 29 Dec 2018 16:44:13 GMT):
but in our code, it boots with a complete list of nodes in network, [here](https://github.com/hyperledger/fabric/blob/d3e80320e5bd97bf7e1232f327be6377c66e3d5f/orderer/consensus/etcdraft/consenter.go#L120)
guoger (Sat, 29 Dec 2018 16:44:36 GMT):
I'll try to push a fix tomorrow
guoger (Sat, 29 Dec 2018 16:44:36 GMT):
created FAB-13456 I'll try to push a fix tomorrow
guoger (Sat, 29 Dec 2018 16:48:08 GMT):
cc @yacovm
yacovm (Sat, 29 Dec 2018 17:18:27 GMT):
thanks @guoger ... I am wondering - how can we tell whether we join an existing network or not? :thinking:
yacovm (Sat, 29 Dec 2018 17:27:07 GMT):
oh, but I think we already do that, no?
```
if c.fresh {
c.logger.Info("starting new raft node")
c.node = raft.StartNode(config, raftPeers)
} else {
c.logger.Info("restarting raft node")
c.node = raft.RestartNode(config)
}
```
guoger (Sat, 29 Dec 2018 17:28:46 GMT):
when a node is started, it's one of:
- fresh new node within fresh cluster
- fresh new node joining existing cluster
- restarted node with some existing data (wal) (and obviously it's part of existing cluster)
guoger (Sat, 29 Dec 2018 17:29:23 GMT):
the code snippet you posted is to differentiate 3) from others
guoger (Sat, 29 Dec 2018 17:29:30 GMT):
however, it doesn't help with 1) and 2)
yacovm (Sat, 29 Dec 2018 17:30:04 GMT):
oh
`[e][OrdererOrg.orderer4] 2018-12-29 14:01:47.025 IST [orderer.consensus.etcdraft] Start -> INFO 134 starting new raft node {"channel": "testchannel2", "node": 4}`
yacovm (Sat, 29 Dec 2018 17:30:38 GMT):
how come it prints this? it shouldn't have the WAL at that point
guoger (Sat, 29 Dec 2018 17:31:10 GMT):
if it has wal, it'll print `Restarting raft node`
yacovm (Sat, 29 Dec 2018 17:31:28 GMT):
oh i see
yacovm (Sat, 29 Dec 2018 17:31:39 GMT):
so, how are we going to solve this then?
guoger (Sat, 29 Dec 2018 17:31:40 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=YpCMiAqH55tsbMkQJ) @yacovm i was wondering the same... and need some input from you :)
yacovm (Sat, 29 Dec 2018 17:31:48 GMT):
hehe
yacovm (Sat, 29 Dec 2018 17:32:00 GMT):
You said you're going to push a fix so i thought you have a solution ;)
yacovm (Sat, 29 Dec 2018 17:32:43 GMT):
In theory we can look at the config block before or something like that, but - i don't understand the etcdraft problem with having a list of peers in the first place... why is that an issue?
guoger (Sat, 29 Dec 2018 17:33:06 GMT):
hehe, that why i put "try" there, cuz i need to read some onboarding work first...
guoger (Sat, 29 Dec 2018 17:33:09 GMT):
if a node is to join a cluster, it should have a config block already?
yacovm (Sat, 29 Dec 2018 17:33:48 GMT):
when you onboard a cluster, you just get the blocks until (including) the latest config block
yacovm (Sat, 29 Dec 2018 17:34:00 GMT):
it's like someone takes your hard drive, puts there the blockchain
yacovm (Sat, 29 Dec 2018 17:34:05 GMT):
and then restarts your OSN node
yacovm (Sat, 29 Dec 2018 17:34:48 GMT):
but i still don't get why the empty list of peers is needed for raft
guoger (Sat, 29 Dec 2018 17:34:50 GMT):
> i don't understand the etcdraft problem with having a list of peers in the first place... why is that an issue?
the short answer is, the way etcdraft is implemented requires this. I could update the jira with a more verbose description tomorrow
guoger (Sat, 29 Dec 2018 17:35:45 GMT):
fairly late here and having an early flight to catch. but i'll keep you posted
yacovm (Sat, 29 Dec 2018 17:36:01 GMT):
> // It appends a ConfChangeAddNode entry for each given peer to the initial log.
I think it's because of this
yacovm (Sat, 29 Dec 2018 17:36:18 GMT):
when you do `StartNode`, it appends your config entry to the log
yacovm (Sat, 29 Dec 2018 17:36:27 GMT):
so you reject the config change of term 2
yacovm (Sat, 29 Dec 2018 17:36:48 GMT):
because I think what you put in the log maybe is different or something
yacovm (Sat, 29 Dec 2018 17:37:05 GMT):
`e := pb.Entry{Type: pb.EntryConfChange, Term: 1, Index: r.raftLog.lastIndex() + 1, Data: d}`
yacovm (Sat, 29 Dec 2018 17:37:11 GMT):
this is what it puts in its log
yacovm (Sat, 29 Dec 2018 17:37:19 GMT):
I see now what's the problem
yacovm (Sat, 29 Dec 2018 17:37:36 GMT):
in Term1 - orderer 4 doesn't exist at all
yacovm (Sat, 29 Dec 2018 17:37:58 GMT):
so when you call `StartNode` the config you persist in the log never happened in the rest of the cluster
yacovm (Sat, 29 Dec 2018 17:38:11 GMT):
since you didn't exist in term 1 for everyone else
yacovm (Sat, 29 Dec 2018 17:39:33 GMT):
I have an idea @guoger - what if we put in `raftPeers := RaftPeers(c.opts.RaftMetadata.Consenters)` all peers but ourselves, in case we are "fresh" ?
yacovm (Sat, 29 Dec 2018 17:39:36 GMT):
can that work?
yacovm (Sat, 29 Dec 2018 17:39:43 GMT):
wdyt?
yacovm (Sat, 29 Dec 2018 17:44:58 GMT):
oh i have an idea... if you don't have a WAL, BUT you have blocks in the ledger, it means you also joined the cluster
guoger (Sat, 29 Dec 2018 17:47:33 GMT):
that's what i thought first, but I wasn't sure if it work in the context of onboarding
yacovm (Sat, 29 Dec 2018 17:47:41 GMT):
let me try and see if it works
guoger (Sat, 29 Dec 2018 17:48:07 GMT):
cool
guoger (Sat, 29 Dec 2018 17:48:11 GMT):
ttyl
yacovm (Sat, 29 Dec 2018 18:51:37 GMT):
ok that didn't go quite well....
yacovm (Sat, 29 Dec 2018 19:14:14 GMT):
so, i don't get this...
`[e][OrdererOrg.orderer4] 2018-12-29 21:00:38.538 IST [orderer.consensus.etcdraft] Start -> INFO 0b2 testchannel starting new raft node {"channel": "testchannel", "node": 7}`
`[e][OrdererOrg.orderer4] 2018-12-29 21:00:38.550 IST [orderer.consensus.etcdraft] Start -> INFO 0c6 systemchannel starting new raft node {"channel": "systemchannel", "node": 7}`
for the system channel and testchannel, this all works well, and it starts a fresh node.
How does it work then, if we said that when you join a node - you need to make it print `restarting raft node` (start with a nil membership) ?
yacovm (Sat, 29 Dec 2018 19:26:37 GMT):
it works!! :woo:
```
Ran 1 of 12 Specs in 74.003 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 11 Skipped
PASS
```
What I did to fix was:
```
//fresh := !wal.Exist(opts.WALDir)
fresh := height == 1
```
yacovm (Sat, 29 Dec 2018 19:58:37 GMT):
While I tested this, what is left to test is what happens when an existing cluster that no node is fresh, is restarted together (shut down all nodes and then bring them all up).
yacovm (Sat, 29 Dec 2018 19:59:10 GMT):
my hypothesis is that in such a case, the state should contain the last membership, and this way it still work.
yacovm (Sat, 29 Dec 2018 19:59:10 GMT):
my hypothesis is that in such a case, the state should contain the last membership, and this way it will still work.
yacovm (Sat, 29 Dec 2018 20:01:11 GMT):
@guoger I'll let you push a change set tomorrow so I can +2 and find another maintainer who is not in Christmas winter hibernation....
guoger (Sat, 29 Dec 2018 23:06:33 GMT):
Sure
guoger (Sun, 30 Dec 2018 00:36:08 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=qmqDEw53gy2Lv5tuc) @yacovm yep, nodes would be _restarting_ in this case, and obtain membership from persisted data.
guoger (Sun, 30 Dec 2018 01:48:33 GMT):
pushed https://gerrit.hyperledger.org/r/28402
guoger (Fri, 04 Jan 2019 06:41:35 GMT):
should be an easy review: https://gerrit.hyperledger.org/r/c/28475/ @yacovm @C0rWin
StefanKosc (Fri, 04 Jan 2019 08:54:51 GMT):
Hello, I tried to install and instantiate two chaincodes concurrently using locust, peer endorses install and instantiation proposals and then it fails on sending instantiate proposal to orderer, does anyone has an idea what might be happening?
StefanKosc (Fri, 04 Jan 2019 08:54:51 GMT):
Hello, I tried to install and instantiate two chaincodes concurrently using locust, peer endorses install and instantiation proposals and then it freezes and then timeouts on sending instantiate proposal to orderer, does anyone has an idea what might be happening?
StefanKosc (Fri, 04 Jan 2019 08:54:51 GMT):
Hello, I tried to install and instantiate two chaincodes concurrently using locust, peer endorses install and instantiation proposals and then it freezes and then timeouts on sending instantiate proposal to orderer, does anyone has an idea what might be happening? the result is that the system breaks and chaincodes cannot be installed and instantiated anymore
StefanKosc (Fri, 04 Jan 2019 11:09:12 GMT):
Hello, I tried to install and instantiate two chaincodes concurrently using locust, sdk freezes after sending instantiate proposals to orderer. In orderer logs it looks that everything is ok and proposals have been endorsed but nothing else happens in sdk, any ideas what might be happening?
kostas (Fri, 04 Jan 2019 21:27:57 GMT):
Hi everyone! As I'm slowly coming back up to speed (reviewing past CRs that have been merged but I missed, etc.), please add me to any pending CRs that you'd like me to have a look at. Thanks!
tock (Sun, 06 Jan 2019 11:00:53 GMT):
@kostas Please take a look at this chain:
https://gerrit.hyperledger.org/r/#/c/27822/
https://gerrit.hyperledger.org/r/#/c/28113/
https://gerrit.hyperledger.org/r/#/c/28155/
https://gerrit.hyperledger.org/r/#/c/28435/
https://gerrit.hyperledger.org/r/#/c/28439/
It covers the consensus-type migration green path end-to-end including integration tests.
@guoger This work may collide with some of the work you do on the etcdraft/chain, so please take note to the developments there, any comment is welcome.
tock (Sun, 06 Jan 2019 11:00:53 GMT):
@kostas Please take a look at this chain:
https://gerrit.hyperledger.org/r/#/c/27822/
https://gerrit.hyperledger.org/r/#/c/28113/
https://gerrit.hyperledger.org/r/#/c/28155/
https://gerrit.hyperledger.org/r/#/c/28435/
https://gerrit.hyperledger.org/r/#/c/28439/
It covers the consensus-type migration green path end-to-end including integration tests.
@guoger This work may collide with some of the work you do on the etcdraft/chain, so please take note to the developments there, any comment is welcome.
tock (Sun, 06 Jan 2019 11:00:53 GMT):
@kostas @jyellick @yacovm @C0rWin Please take a look at this chain:
https://gerrit.hyperledger.org/r/#/c/27822/
https://gerrit.hyperledger.org/r/#/c/28113/
https://gerrit.hyperledger.org/r/#/c/28155/
https://gerrit.hyperledger.org/r/#/c/28435/
https://gerrit.hyperledger.org/r/#/c/28439/
It covers the consensus-type migration green path end-to-end including integration tests.
@guoger This work may collide with some of the work you do on the etcdraft/chain, so please take note to the developments there, any comment is welcome.
tock (Sun, 06 Jan 2019 11:02:52 GMT):
Note that the respective JIRA items contain a more detailed description than what is included in the commit message, so look there when you review.
guoger (Mon, 07 Jan 2019 09:35:55 GMT):
working on [slides](https://docs.google.com/presentation/d/1H_aajW2mDsKa8Q-mayvEI-p2brl4O0jRsfcC6O8t2rg/edit?usp=sharing) for community playback. pls give it a look and leave some comments if you find anything missing/inaccurate/redundant, thx in advance!
yacovm (Mon, 07 Jan 2019 09:39:49 GMT):
I think we should add a slide about the fact that nodes identify each other by their TLS certificates
yacovm (Mon, 07 Jan 2019 09:40:01 GMT):
and stress out that you have to run EtcdRaft OSNs only with TLS
yacovm (Mon, 07 Jan 2019 09:40:49 GMT):
And to explain that we prevent a sybil attack of connection foreign OSNs by having the OSNs authenticate each other via TLS pinning
yacovm (Mon, 07 Jan 2019 09:41:14 GMT):
Also instead of `Replicator` let's call this `Replication`
yacovm (Mon, 07 Jan 2019 09:42:22 GMT):
and I wouldn't put the slide of the interface of the gRPC layer in the slides
yacovm (Mon, 07 Jan 2019 09:42:31 GMT):
community cares about functionality, less about implementation IMO
C0rWin (Mon, 07 Jan 2019 09:46:20 GMT):
a nit, but I would not put `channel C` on slide 4 with two nodes only
C0rWin (Mon, 07 Jan 2019 09:47:01 GMT):
> and stress out that you have to run EtcdRaft OSNs only with TLS
to add to it, there are two TLS certificates, for cluster communication and for clients
C0rWin (Mon, 07 Jan 2019 09:47:46 GMT):
Clipboard - January 7, 2019 11:47 AM
C0rWin (Mon, 07 Jan 2019 09:49:07 GMT):
this diagram very confusing, since it makes think that OSNs consumes from kafka different transactions, while they are using kafka topic per channel and within the channel there is offsets per blocks
guoger (Mon, 07 Jan 2019 09:51:17 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=jwWA6hZdMgABmujHQ) @C0rWin because of different color on txs?
C0rWin (Mon, 07 Jan 2019 09:52:24 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=fkeYH4mTe8GfBCwJY) @guoger yes, I'm guessing you wanted to show that orderes using topic per channel. but it look different
guoger (Mon, 07 Jan 2019 09:54:01 GMT):
huh, actually i did not intend to indicate anything related to channels... (put differently, that diagram only shows one channel)
guoger (Mon, 07 Jan 2019 09:55:09 GMT):
txs are put in different color only for visual effect. but clearly it already causes confusion :P
guoger (Mon, 07 Jan 2019 09:55:18 GMT):
maybe in the same color? wdyt @C0rWin
C0rWin (Mon, 07 Jan 2019 10:42:51 GMT):
same color will be fine, I guess
C0rWin (Mon, 07 Jan 2019 10:42:51 GMT):
same color will be fine, I guess @guoger
C0rWin (Mon, 07 Jan 2019 10:48:15 GMT):
when is the community playback? @guoger
yacovm (Mon, 07 Jan 2019 11:26:22 GMT):
@guoger one more thing :)
instead of writing "reusable" I think it's better phrased as "independent of consensus plugin", what do you think?
adarshsaraf123 (Mon, 07 Jan 2019 11:27:41 GMT):
maybe i have missed some details in the past, but did we enforce the system channel to be run on all OSNs? Can an OSN not be part of the system channel and still serve another channel?
yacovm (Mon, 07 Jan 2019 11:28:44 GMT):
all OSNs run the system channel. Otherwise they can't know about channel creation.
an application channel can service only a subset of OSNs
yacovm (Mon, 07 Jan 2019 11:28:44 GMT):
all OSNs run the system channel. Otherwise they can't know about channel creation.
an application channel can be serviced by only a subset of OSNs
guoger (Mon, 07 Jan 2019 13:03:28 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=9njQBa5jWu2hP54yg) @C0rWin it's 9am-10am EST
kostas (Mon, 07 Jan 2019 13:23:18 GMT):
https://wiki.hyperledger.org/projects/fabric/playbacks
yacovm (Mon, 07 Jan 2019 21:04:12 GMT):
What is going on (here)[https://jenkins.hyperledger.org/job/fabric-verify-integration-tests-x86_64/4481/consoleFull] guys?
It seems like a transaction was sent but never reached any peer. `20:50:13 [e][peer-chaincode-invoke] Error: timed out waiting for txid on all peers - proposal response:
yacovm (Mon, 07 Jan 2019 21:04:12 GMT):
What is going on here- https://jenkins.hyperledger.org/job/fabric-verify-integration-tests-x86_64/4481/consoleFull guys?
It seems like a transaction was sent but never reached any peer. `20:50:13 [e][peer-chaincode-invoke] Error: timed out waiting for txid on all peers - proposal response:
tock (Mon, 07 Jan 2019 21:27:29 GMT):
@yacovm can you take a look at this? gossip is flaky in unit tests, is this a known problem?
tock (Mon, 07 Jan 2019 21:27:34 GMT):
https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/7674/console
yacovm (Mon, 07 Jan 2019 21:35:44 GMT):
yep
guoger (Tue, 08 Jan 2019 06:08:01 GMT):
@yacovm there *is* a leader (3), however its _heartbeats_ do not reach (4) in timely fashion. while (4) is campaigning for leadership, it receives a transaction, and forwards it to (3). Then (4) got elected as new leader. That tx never got committed. Since it is forwarded by follower, client is not notified
guoger (Tue, 08 Jan 2019 06:10:12 GMT):
the problem is, as log indicates, *(3)* can receive step messages (it acknowledges new leader *(4)*), but not sending step messages out (*(2)* doesn't receive heartbeats neither).
guoger (Tue, 08 Jan 2019 06:10:12 GMT):
the problem is, as log indicates, *(3)* can receive step messages (it acknowledges new leader *(4)*), but not sending step messages out ( *(2)* doesn't receive heartbeats neither).
yacovm (Tue, 08 Jan 2019 07:46:19 GMT):
Why is client not notified?
yacovm (Tue, 08 Jan 2019 07:46:35 GMT):
The notification comes from a perr
yacovm (Tue, 08 Jan 2019 07:46:38 GMT):
*peer
guoger (Tue, 08 Jan 2019 09:03:24 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=8d29bf0e-c01e-4f37-ac05-dc48c6fc5133) @yacovm by client, i mean whoever submits that tx to orderer. It is not notified because that orderer actually successfully forwards tx to leader (and leader actually receives it)
yacovm (Tue, 08 Jan 2019 09:03:51 GMT):
if it transfers the transaction to the leader then the transaction should be committed
yacovm (Tue, 08 Jan 2019 09:03:53 GMT):
but it is never committed
yacovm (Tue, 08 Jan 2019 09:03:59 GMT):
that's the error in the test...
guoger (Tue, 08 Jan 2019 09:07:50 GMT):
i guess i was just trying to say, the test failed not because there's no leader, but because leader does not send out step messages. As a matter of why that happened, i haven't figured out yet.
yacovm (Tue, 08 Jan 2019 09:11:39 GMT):
if the leader's heartbeats don't reach (4) then it can't forward to the leader, right? because it doesn't know a leader
yacovm (Tue, 08 Jan 2019 09:12:02 GMT):
if it doesn't know a leader, and it is not a leader itself - it needs to return a bad status of unavailable to the submit attempt from the client
guoger (Tue, 08 Jan 2019 09:15:19 GMT):
think it as one-way pipe
guoger (Tue, 08 Jan 2019 09:15:35 GMT):
it *does* know a leader
yacovm (Tue, 08 Jan 2019 09:15:45 GMT):
ah... i see
yacovm (Tue, 08 Jan 2019 09:16:52 GMT):
so, it forwards the message to the leader then... but it never gets into any block
guoger (Tue, 08 Jan 2019 09:17:08 GMT):
i suspect that wal sync took too long,
> [e][OrdererOrg.orderer3] 2019-01-07 18:49:42.154059 W | wal: sync duration of 1.563002974s, expected less than 1s
and leader was stuck
guoger (Tue, 08 Jan 2019 09:17:08 GMT):
i suspect that wal sync took too long,
> [e][OrdererOrg.orderer3] 2019-01-07 18:49:42.154059 W | wal: sync duration of 1.563002974s, expected less than 1s
and leader was stuck beyond election timeout
yacovm (Tue, 08 Jan 2019 09:17:19 GMT):
hmmm
yacovm (Tue, 08 Jan 2019 09:17:36 GMT):
i see...
yacovm (Tue, 08 Jan 2019 09:17:55 GMT):
but it's only 1.5 seconds
guoger (Tue, 08 Jan 2019 09:18:04 GMT):
however, i'll need to give it another look... after playback maybe
yacovm (Tue, 08 Jan 2019 09:18:10 GMT):
ok ok
guoger (Tue, 08 Jan 2019 09:18:13 GMT):
election timeout is 1 second
guoger (Tue, 08 Jan 2019 09:18:27 GMT):
100ms * 10
yacovm (Tue, 08 Jan 2019 09:18:48 GMT):
election timeout is the time of heart-beats?
guoger (Tue, 08 Jan 2019 09:19:02 GMT):
no, by default, it's 10 times of heartbeat
yacovm (Tue, 08 Jan 2019 09:19:13 GMT):
got it
guoger (Tue, 08 Jan 2019 09:22:41 GMT):
actually, if you happen to know this (which is very likely :P), how am i supposed to use `configtxlator` to create a tx that adds a consenter? my current steps are:
- fetch `genesis.block`
- edit `configtx.yaml` to contain one extra consenter, and use `configtxgen` to produce `genesis.udpated.block`
- `configtxlator compute_update` those two genesis blocks
However I got `no differences detected between original and updated config`.
guoger (Tue, 08 Jan 2019 09:23:28 GMT):
if I decode them to json and diff, there's difference. @yacovm
yacovm (Tue, 08 Jan 2019 09:27:28 GMT):
just look at my script
guoger (Tue, 08 Jan 2019 09:27:51 GMT):
script?
yacovm (Tue, 08 Jan 2019 09:28:08 GMT):
https://gerrit.hyperledger.org/r/#/c/27734/4/examples/e2e_cli/scripts/script.sh@76
tock (Tue, 08 Jan 2019 12:04:55 GMT):
@yacovm gossip unit test is punishing me again: https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/7686/console
I think this defies the purpose of unit tests... if 1 out of 3 runs succeeds, people stop looking at these tests results, not to mention the waste of time and resources. I think it is worth identifying the flaky test cases and correcting them or removing them for CI builds.
yacovm (Tue, 08 Jan 2019 14:03:44 GMT):
@tock not sure what is your point?
yacovm (Tue, 08 Jan 2019 14:03:59 GMT):
this isn't related to #fabric-orderer-dev :)
tock (Tue, 08 Jan 2019 14:29:39 GMT):
@yacovm my point is, someone has got to do it, and you are the best man for the job ;-) (or maybe @C0rWin can do it better then you? :yum:)
C0rWin (Tue, 08 Jan 2019 14:30:52 GMT):
@tock we are working on this 24/7
yacovm (Tue, 08 Jan 2019 14:31:04 GMT):
but I am the best man for any job @tock , not sure what's your point :/
yacovm (Tue, 08 Jan 2019 14:33:33 GMT):
@C0rWin can fix it if he wants, I don't have any feelings toward gossip
kostas (Tue, 08 Jan 2019 16:18:38 GMT):
@tock: Not sure if a new patchset was to be submitted here: https://gerrit.hyperledger.org/r/c/28435/
kostas (Tue, 08 Jan 2019 16:18:38 GMT):
@tock: Not sure if a new patchset was to be submitted here? https://gerrit.hyperledger.org/r/c/28435/
kostas (Tue, 08 Jan 2019 16:18:53 GMT):
(I see items marked as Done on the most recent patchset.)
kostas (Tue, 08 Jan 2019 16:19:10 GMT):
I'll keep reviewing PS6.
tock (Wed, 09 Jan 2019 06:37:36 GMT):
Not submitted yet. I am working on you're reviews and commenting as I fix it locally. Will let you know when I push a new one. Thanks for the review!
tock (Wed, 09 Jan 2019 06:37:36 GMT):
Not submitted yet. I am working on your reviews and commenting as I fix it locally. Will let you know when I push a new one. Thanks for the review!
tock (Wed, 09 Jan 2019 06:37:36 GMT):
@kostas Not submitted yet. I am working on your reviews and commenting as I fix it locally. Will let you know when I push a new one. Thanks for the review!
yacovm (Wed, 09 Jan 2019 07:33:15 GMT):
@guoger , I think we should do [FAB-13438](https://jira.hyperledger.org/browse/FAB-13438), otherwise we diverge from the user experience kafka OSNs give us - return service unavailable when the consensus is faulty.
yacovm (Wed, 09 Jan 2019 07:33:42 GMT):
You commented in that JIRA but that was something not really related to what I advocate for :)
yacovm (Wed, 09 Jan 2019 07:34:21 GMT):
I infer from gerrit that you have enough things on your plate, so I can just take on this, and push a CR either rebased on top of master or on top of some CR of your choice
yacovm (Wed, 09 Jan 2019 07:34:23 GMT):
what do you say?
yacovm (Wed, 09 Jan 2019 07:36:18 GMT):
(It's not like I really want to touch the chain's code, particularly the ginkgo tests(...) but I think that since SVT has more or less started, we should try and close down any functional-altering items as soon as possible)
yacovm (Wed, 09 Jan 2019 07:37:09 GMT):
(of course, if you want to take this instead - I'm also good with that)
yacovm (Wed, 09 Jan 2019 07:37:58 GMT):
But I think it should be done, by someone, one way or another... we can't have an OSN be disconnected from the cluster and return on Deliver that everything is great
guoger (Wed, 09 Jan 2019 11:23:45 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=8XrSiuCyZxfS2vzro) @yacovm what's the delta?
yacovm (Wed, 09 Jan 2019 11:37:50 GMT):
the delta is 100%
yacovm (Wed, 09 Jan 2019 11:37:58 GMT):
I'm not talking about leader stepping down
guoger (Wed, 09 Jan 2019 11:39:09 GMT):
to realize "return service unavailable when the consensus is faulty", you'll need to detect "faulty", right? :)
yacovm (Wed, 09 Jan 2019 11:40:07 GMT):
but we can just probe the soft state no?
yacovm (Wed, 09 Jan 2019 11:40:51 GMT):
I mean - if the soft state returns there is no leader
yacovm (Wed, 09 Jan 2019 11:41:02 GMT):
then we should return service unavailable in response to deliver requests
yacovm (Wed, 09 Jan 2019 11:41:13 GMT):
what we do now, is - we actually serve the block to the client
yacovm (Wed, 09 Jan 2019 11:41:16 GMT):
but we shouldn't do that
yacovm (Wed, 09 Jan 2019 11:41:27 GMT):
we should instead make the client go find some other orderer
guoger (Wed, 09 Jan 2019 11:43:07 GMT):
a node that _already_ knows there's no leader?
yacovm (Wed, 09 Jan 2019 11:43:27 GMT):
yeah
guoger (Wed, 09 Jan 2019 11:50:13 GMT):
that part is fine, but you couldn't really tell whether there's leader or not w/o `CheckQuorum`, right?
yacovm (Wed, 09 Jan 2019 11:50:30 GMT):
why not?
yacovm (Wed, 09 Jan 2019 11:50:33 GMT):
don't we do this now?
yacovm (Wed, 09 Jan 2019 11:50:42 GMT):
when you send a submit RPC
yacovm (Wed, 09 Jan 2019 11:50:48 GMT):
you check who is the leader
yacovm (Wed, 09 Jan 2019 11:50:57 GMT):
if it's 0, you return an error back, right>
yacovm (Wed, 09 Jan 2019 11:50:57 GMT):
if it's 0, you return an error back, right?
guoger (Wed, 09 Jan 2019 11:51:20 GMT):
let me rephrase, as an ex-leader, you don't know when to step down, unless somebody else is elected (as the code currently stands)
yacovm (Wed, 09 Jan 2019 11:51:50 GMT):
you're saying that we have a problem of false positive
yacovm (Wed, 09 Jan 2019 11:52:01 GMT):
but it doesn't mean that there are cases where you know there is no leader
yacovm (Wed, 09 Jan 2019 11:52:11 GMT):
your proposal is for leaders that are no longer leaders
yacovm (Wed, 09 Jan 2019 11:52:21 GMT):
I am talking here about a node, in general- leader or not
guoger (Wed, 09 Jan 2019 11:53:38 GMT):
w/o `CheckQuorum`, "no Raft leader" only happens when cluster has just started
yacovm (Wed, 09 Jan 2019 11:53:57 GMT):
really?
yacovm (Wed, 09 Jan 2019 11:54:11 GMT):
even for nodes that were never leaders?
guoger (Wed, 09 Jan 2019 12:01:23 GMT):
if a follower doesn't receive heartbeats in timeout, it becomes pre-candidate and start campaigning. in this state, it still recognizes the leader (forwarding tx to it), until it actually becomes candidate
guoger (Wed, 09 Jan 2019 12:01:23 GMT):
if a follower doesn't receive heartbeats in timeout, it becomes pre-candidate and start pre-voting. in this state, it still recognizes the leader (forwarding tx to it), until it actually becomes candidate
yacovm (Wed, 09 Jan 2019 12:01:57 GMT):
I don't think this is good.... do you?
guoger (Wed, 09 Jan 2019 12:02:08 GMT):
why not?
yacovm (Wed, 09 Jan 2019 12:02:23 GMT):
because what happens to the envelope it forwarded to the leader which is not the leader anymore?
guoger (Wed, 09 Jan 2019 12:02:55 GMT):
because as a follower, without receiving heartbeats, you don't know if yourself is disconnected, or leader is disconnected
guoger (Wed, 09 Jan 2019 12:03:19 GMT):
so you start probing, and find out
guoger (Wed, 09 Jan 2019 12:03:31 GMT):
either yourself can be elected, or you are disconnected
yacovm (Wed, 09 Jan 2019 12:03:41 GMT):
it doesn't matter if it's your fault or the leader's fault - I say that you should return an error back to the user if the leader didn't ping you within a timely manner
yacovm (Wed, 09 Jan 2019 12:04:11 GMT):
else the envelope the user sent you will be silently discarded
yacovm (Wed, 09 Jan 2019 12:04:27 GMT):
this way at least you give the client a chance to re-transmit the transaction to another OSN
guoger (Wed, 09 Jan 2019 12:16:30 GMT):
i agree that we should return error if we *know* there's no leader. however, we simple are not able to make any guarantees. (packets that are lost on wire are "silently dropped" too).
There's one case I felt that we should do something about, where follower is disconnected, and just keeps campaigning in vein. It still recognizes leader in softstate.
guoger (Wed, 09 Jan 2019 12:16:44 GMT):
etcdraft doesn't help us there (CheckQuorum is only for leader)
guoger (Wed, 09 Jan 2019 12:17:22 GMT):
however, i'd say shouldn't `SendSubmit` call return error in this case? (tcp link is broken)
yacovm (Wed, 09 Jan 2019 12:17:46 GMT):
if packets are lost they are retransmitted because TCP ensures FIFO delivery or an error report
yacovm (Wed, 09 Jan 2019 12:18:05 GMT):
if you send down a gRPC stream 2 messages, they will never reach in an opposite order
guoger (Wed, 09 Jan 2019 12:18:42 GMT):
therefore, packet won't dropped "silently", right?
yacovm (Wed, 09 Jan 2019 12:18:49 GMT):
> however, i'd say shouldn't `SendSubmit` call return error in this case? (tcp link is broken)
so, SendSubmit is asynchronous
yacovm (Wed, 09 Jan 2019 12:18:59 GMT):
you just put a message down a buffer
yacovm (Wed, 09 Jan 2019 12:19:06 GMT):
gRPC http/2 transport does the transfer
yacovm (Wed, 09 Jan 2019 12:19:20 GMT):
if the underlying http/2 layer reaches a conclusion that the stream is broken
yacovm (Wed, 09 Jan 2019 12:19:23 GMT):
then it returns an error
yacovm (Wed, 09 Jan 2019 12:19:59 GMT):
I have a question
yacovm (Wed, 09 Jan 2019 12:20:03 GMT):
I don't understand something
yacovm (Wed, 09 Jan 2019 12:20:09 GMT):
> etcdraft doesn't help us there (CheckQuorum is only for leader)
yacovm (Wed, 09 Jan 2019 12:20:28 GMT):
does it mean that if you're a follower and you start campaign for a leader election because you concluded the leader is down
yacovm (Wed, 09 Jan 2019 12:20:46 GMT):
but you never, ever, say- within an hour - make contact with the rest of the cluster
yacovm (Wed, 09 Jan 2019 12:20:56 GMT):
so you effectively have *no leader* in an hour
yacovm (Wed, 09 Jan 2019 12:21:00 GMT):
my question is....
yacovm (Wed, 09 Jan 2019 12:21:09 GMT):
what does the soft state return w.r.t the leader number?
yacovm (Wed, 09 Jan 2019 12:21:16 GMT):
is it 0, or the previous leader ID?
guoger (Wed, 09 Jan 2019 12:21:35 GMT):
previous leader ID
yacovm (Wed, 09 Jan 2019 12:21:38 GMT):
oh my god
yacovm (Wed, 09 Jan 2019 12:21:48 GMT):
this is horrible
yacovm (Wed, 09 Jan 2019 12:22:08 GMT):
it means if we have an internal cluster issue
yacovm (Wed, 09 Jan 2019 12:22:23 GMT):
and you select randomly OSNs
yacovm (Wed, 09 Jan 2019 12:22:57 GMT):
and the internal cluster network is down
yacovm (Wed, 09 Jan 2019 12:23:09 GMT):
or, say - you misconfigured the certificates
yacovm (Wed, 09 Jan 2019 12:23:24 GMT):
then with probability of (n-1)/n - your transaction goes to /dev/null :/
yacovm (Wed, 09 Jan 2019 12:23:33 GMT):
and you don't know about it...
guoger (Wed, 09 Jan 2019 12:24:07 GMT):
haha, it's n/n, because even you get to leader, your tx is not consented
yacovm (Wed, 09 Jan 2019 12:24:16 GMT):
the leader has the check quorum
yacovm (Wed, 09 Jan 2019 12:24:20 GMT):
so it can return an error
yacovm (Wed, 09 Jan 2019 12:24:35 GMT):
at least you know, things are bad
yacovm (Wed, 09 Jan 2019 12:25:03 GMT):
anyway.... I need to go now, be back later.... I suggest @kostas reads this conversation when he wakes up
yacovm (Wed, 09 Jan 2019 12:25:12 GMT):
and we can discuss later here in this channel, or tomorrow on scrum
guoger (Wed, 09 Jan 2019 12:25:41 GMT):
sure
kostas (Wed, 09 Jan 2019 15:16:31 GMT):
> Not submitted yet. I am working on your reviews and commenting as I fix it locally.
@tock: I hear you. Let's do this instead. Comment locally for your own book-keeping but please do not submit until after your patchset is pushed. Gerrit persists your comments across sessions (even if you close your tab/do not submit the review). This makes bookkeeping easier for the reviewer as well, and is the established norm.
kostas (Wed, 09 Jan 2019 15:28:13 GMT):
``` // CheckQuorum specifies if the leader should check quorum activity. Leader
// steps down when quorum is not active for an electionTimeout.
CheckQuorum bool```
kostas (Wed, 09 Jan 2019 15:28:13 GMT):
```
// CheckQuorum specifies if the leader should check quorum activity. Leader
// steps down when quorum is not active for an electionTimeout.
CheckQuorum bool
```
kostas (Wed, 09 Jan 2019 15:29:01 GMT):
First of all, I see this referenced in the discussion and I'm not sure I get how it's connected?
kostas (Wed, 09 Jan 2019 15:30:41 GMT):
What is quorum activity? Transmission and reception of heartbeats?
kostas (Wed, 09 Jan 2019 15:30:44 GMT):
Something else?
kostas (Wed, 09 Jan 2019 15:32:56 GMT):
The one safe conclusion I gather from the discussion is that we cannot rely on `SoftState.Lead` to tell whether a channel is leaderless. (We were hoping it'd be 0, but it isn't.)
kostas (Wed, 09 Jan 2019 15:34:03 GMT):
And all incoming transactions during this leaderless state get forwarded to the current value of `SoftState.Lead` (which is not accurate, if we're leaderless).
kostas (Wed, 09 Jan 2019 15:34:15 GMT):
So these transactions are lost.
kostas (Wed, 09 Jan 2019 15:35:23 GMT):
Assuming `CheckQuorum` is set to `true` --
kostas (Wed, 09 Jan 2019 15:35:49 GMT):
Even if the leader steps down, we still cannot rely on `SoftState.Lead` (since it would carry that node's ID).
kostas (Wed, 09 Jan 2019 15:36:01 GMT):
So, forget everything else for a second.
kostas (Wed, 09 Jan 2019 15:36:40 GMT):
Assuming we only care about the former leader saying "we don't have a leader!". How do we do this @guoger? What is the programmatic check?
guoger (Wed, 09 Jan 2019 15:38:28 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=APjcwKZmRchr7hpsS) @kostas Enable `CheckQuorum`, former leader will step down when it doesn't receive MsgHeartbeatResp from quorum
guoger (Wed, 09 Jan 2019 15:39:20 GMT):
it steps down, and sets it to zero
kostas (Wed, 09 Jan 2019 15:39:32 GMT):
Awesome.
kostas (Wed, 09 Jan 2019 15:40:04 GMT):
So let's go back to the problem that Yacov is referring to.
kostas (Wed, 09 Jan 2019 15:40:37 GMT):
He's suggesting that we return service unavailable when the channel is leaderless (basically).
kostas (Wed, 09 Jan 2019 15:41:01 GMT):
And we said the most recent leader can do this check by himself.
guoger (Wed, 09 Jan 2019 15:41:49 GMT):
yes
kostas (Wed, 09 Jan 2019 15:42:01 GMT):
Can we have the followers periodically ping the most recent leader (i.e. the value in their `SoftState.Lead`?) to get that value back?
kostas (Wed, 09 Jan 2019 15:42:24 GMT):
Either by piggybacking on an existing RPC, or by -gasp- creating a new one?
kostas (Wed, 09 Jan 2019 15:43:25 GMT):
On a side note: No matter how we go about approaching, this will truly kill our throughput.
kostas (Wed, 09 Jan 2019 15:43:54 GMT):
I'm kind of surprised that etcd/raft is so laissez-faire about this as well, TBH and this is something I had not picked up on when I was exploring it.
kostas (Wed, 09 Jan 2019 15:43:54 GMT):
I'm kind of surprised that etcd/raft is so laissez-faire about this as well TBH and this is something I had not picked up on when I was exploring it.
guoger (Wed, 09 Jan 2019 15:44:40 GMT):
> to get that value back?
which value?
kostas (Wed, 09 Jan 2019 15:44:55 GMT):
> it steps down, and sets it to zero
kostas (Wed, 09 Jan 2019 15:45:16 GMT):
My understanding is that the (former) leader's `SoftState.Lead` value is the absolute source of truth.
kostas (Wed, 09 Jan 2019 15:45:38 GMT):
Well, I can already see how this is not true that I typed it.
guoger (Wed, 09 Jan 2019 15:46:58 GMT):
what we are really trying to solve is this:
> if a node is in `Candidate` state, it should return service unavailable
kostas (Wed, 09 Jan 2019 15:47:34 GMT):
Oh, I see the equivalence.
kostas (Wed, 09 Jan 2019 15:47:46 GMT):
And your take is?
kostas (Wed, 09 Jan 2019 15:48:02 GMT):
What prevents us from checking our state and returning the appropriate status?
guoger (Wed, 09 Jan 2019 15:53:30 GMT):
nothing.. we just don't `RaftState` in `SoftState` today. we can just make use of it, and solve this problem
guoger (Wed, 09 Jan 2019 15:53:30 GMT):
nothing.. we are just not using `RaftState` in `SoftState` today. we can just make use of it, and solve this problem
guoger (Wed, 09 Jan 2019 15:54:37 GMT):
instead of using `Lead` in `SoftState`, we turn to use `RaftState ` to determine status.
kostas (Wed, 09 Jan 2019 15:54:45 GMT):
Perfect.
kostas (Wed, 09 Jan 2019 15:55:11 GMT):
@yacovm: Does this address the concern?
tock (Wed, 09 Jan 2019 16:08:35 GMT):
@kostas @C0rWin @yacovm Unless there are further comments, I think these two are ready to merge: https://gerrit.hyperledger.org/r/#/c/28113/ https://gerrit.hyperledger.org/r/#/c/28155/
tock (Wed, 09 Jan 2019 16:09:33 GMT):
I pushed a big patch that addressed your comments on this one, please review: https://gerrit.hyperledger.org/r/#/c/28435/
yacovm (Wed, 09 Jan 2019 16:20:14 GMT):
enabling check quorum won't help
yacovm (Wed, 09 Jan 2019 16:20:34 GMT):
the leader might step down but the followers might not see that if they are disconnected to it
yacovm (Wed, 09 Jan 2019 16:23:52 GMT):
I don't think we should have the followers probe the leader
yacovm (Wed, 09 Jan 2019 16:24:05 GMT):
we can just intercept the hearbeats from the leader instead
yacovm (Wed, 09 Jan 2019 16:24:09 GMT):
since it already does that
yacovm (Wed, 09 Jan 2019 16:24:19 GMT):
but we need to decide on how to approach this
yacovm (Wed, 09 Jan 2019 16:24:34 GMT):
*and* make sure clients are aware their transaction can not be safely forwarded to the leadwr
kostas (Wed, 09 Jan 2019 16:31:05 GMT):
> I don't think we should have the followers probe the leader
kostas (Wed, 09 Jan 2019 16:31:31 GMT):
We agreed that we won't do that and instead inspect our own `RaftState`, no?
yacovm (Wed, 09 Jan 2019 16:39:16 GMT):
wait, so `RaftState` is up to date or something?
yacovm (Wed, 09 Jan 2019 16:39:41 GMT):
what is the status of raft state?
kostas (Wed, 09 Jan 2019 16:40:48 GMT):
If a node senses that there's no leader, it won't be reflected in `SoftState.Lead` but it will be reflected in the fact they're campaigning.
yacovm (Wed, 09 Jan 2019 16:41:25 GMT):
did you verify it ?
guoger (Wed, 09 Jan 2019 16:41:29 GMT):
i did
yacovm (Wed, 09 Jan 2019 16:41:51 GMT):
I think we should add a UT for this, honestly
yacovm (Wed, 09 Jan 2019 16:41:58 GMT):
if we update the etcdraft package
yacovm (Wed, 09 Jan 2019 16:42:01 GMT):
and it somehow breaks
yacovm (Wed, 09 Jan 2019 16:42:07 GMT):
we don't know this otherwise
guoger (Wed, 09 Jan 2019 16:43:15 GMT):
if we reach the conclusion that this is the way to go, i'll surely add UT while working on this
guoger (Wed, 09 Jan 2019 16:43:40 GMT):
and actually FAB-13438 is the next thing to tackle on my side, as i promised to kostas at the beginning of this week :P
guoger (Wed, 09 Jan 2019 16:43:57 GMT):
especially when i just pushed CRs for FAB-13447
yacovm (Wed, 09 Jan 2019 17:15:43 GMT):
ok... so do you guys agree to the idea of sending service unavailable to deliver requests, once there is no leader, + periodically probing the raft state?
guoger (Wed, 09 Jan 2019 17:41:45 GMT):
I agree, other than the part "periodically probing the raft state".
guoger (Wed, 09 Jan 2019 17:41:59 GMT):
there's not need to probe it
guoger (Wed, 09 Jan 2019 17:41:59 GMT):
there's no need to probe it
yacovm (Wed, 09 Jan 2019 17:55:50 GMT):
why not?
yacovm (Wed, 09 Jan 2019 17:56:06 GMT):
how do you know if you suddenly lost the leader?
guoger (Wed, 09 Jan 2019 17:56:55 GMT):
if follower doesn't receive heartbeat in timeout, it loses leader
guoger (Wed, 09 Jan 2019 17:57:32 GMT):
we just inspect raft state, but why periodically?
yacovm (Wed, 09 Jan 2019 17:58:01 GMT):
because you need to tell the clients connected to you via deliver
yacovm (Wed, 09 Jan 2019 17:58:05 GMT):
to look somewhere else
guoger (Wed, 09 Jan 2019 17:58:58 GMT):
isn't that equivalent to returning a closed channel on `Errored()`?
yacovm (Wed, 09 Jan 2019 18:29:43 GMT):
@guoger but we don't do that
yacovm (Wed, 09 Jan 2019 19:23:55 GMT):
@kostas ...
> ok... so do you guys agree to the idea of sending service unavailable to deliver requests, once there is no leader, + periodically probing the raft state?
?
kostas (Wed, 09 Jan 2019 19:24:36 GMT):
I do, and saw the discussion but I've been meaning to go back and check what `Errored` does. (Caught in a couple of other threads at the moment.) Unless you have it handy?
kostas (Wed, 09 Jan 2019 19:31:45 GMT):
> isn't that equivalent to returning a closed channel on `Errored()`?
Reading the description for `Errored` in `consensus.go` this is exactly what I would expect us to do.
yacovm (Wed, 09 Jan 2019 19:32:32 GMT):
it doesn't do what the kafka does
kostas (Wed, 09 Jan 2019 19:32:41 GMT):
Ah, I see the code now.
yacovm (Wed, 09 Jan 2019 19:32:42 GMT):
in etcdraft it returns the ready channel....
kostas (Wed, 09 Jan 2019 19:32:50 GMT):
`// Errored returns a channel that closes when the chain stops.
func (c *Chain) Errored() <-chan struct{} {
return c.doneC
}
`
kostas (Wed, 09 Jan 2019 19:32:50 GMT):
```
// Errored returns a channel that closes when the chain stops.
func (c *Chain) Errored() <-chan struct{} {
return c.doneC
}
```
yacovm (Wed, 09 Jan 2019 19:32:55 GMT):
yeah
yacovm (Wed, 09 Jan 2019 19:32:59 GMT):
we need to be uniform :)
yacovm (Wed, 09 Jan 2019 19:33:29 GMT):
so a peer/user would be oblivious of the type of OSN
kostas (Wed, 09 Jan 2019 19:33:33 GMT):
Why don't we modify `Errored` so that it works like we expect it to?
kostas (Wed, 09 Jan 2019 19:33:39 GMT):
Or is that you've been proposing all along?
yacovm (Wed, 09 Jan 2019 19:33:52 GMT):
we need to couple that with a periodical probing no?
yacovm (Wed, 09 Jan 2019 19:34:02 GMT):
otherwise you don't know when to close the channel
kostas (Wed, 09 Jan 2019 19:34:20 GMT):
Is it not enough to close if you're a candidate?
kostas (Wed, 09 Jan 2019 19:34:35 GMT):
Doesn't this mean that the king is dead, and we have a Game of Thrones thing going on?
yacovm (Wed, 09 Jan 2019 19:34:47 GMT):
it is (you don't want to close the ready channel, right? you need to close another channel)
yacovm (Wed, 09 Jan 2019 19:34:50 GMT):
but the point it
yacovm (Wed, 09 Jan 2019 19:34:56 GMT):
you need to know *when* to close
yacovm (Wed, 09 Jan 2019 19:35:07 GMT):
and we currently can't do that because we don't have periodic probing
yacovm (Wed, 09 Jan 2019 19:35:29 GMT):
or if we do that while piggybacking on the FSM tick
yacovm (Wed, 09 Jan 2019 19:35:34 GMT):
that's also an option
yacovm (Wed, 09 Jan 2019 19:35:49 GMT):
but I don't like that because it's too "frequent"
kostas (Wed, 09 Jan 2019 19:36:22 GMT):
I'm saying the following: Create a channel whenever we're not a candidate. Close that channel when we become a candidate. Return that channel via Errored. What am I missing?
yacovm (Wed, 09 Jan 2019 19:36:36 GMT):
> Doesn't this mean that the king is dead, and we have a Game of Thrones thing going on?
I honestly never ever watched a single episode of GoT
yacovm (Wed, 09 Jan 2019 19:36:50 GMT):
I guess you realize, I don't have time for this
kostas (Wed, 09 Jan 2019 19:36:57 GMT):
It's a family-friendly show where absolutely nobody dies.
yacovm (Wed, 09 Jan 2019 19:37:09 GMT):
I know the content :)
kostas (Wed, 09 Jan 2019 19:37:19 GMT):
Back to the question at hand, what am I missing?
yacovm (Wed, 09 Jan 2019 19:38:23 GMT):
what you propose here:
> I'm saying the following: Create a channel whenever we're not a candidate. Close that channel when we become a candidate. Return that channel via Errored. What am I missing?
seems what I had in mind. The question is - how, or - when do you close the channel when you become a candidate?
yacovm (Wed, 09 Jan 2019 19:38:40 GMT):
there is no callback that the `raft.Node` calls you to do that
kostas (Wed, 09 Jan 2019 19:38:47 GMT):
Ah, I get you now.
kostas (Wed, 09 Jan 2019 19:38:59 GMT):
This is where I point my finger to Jay.
kostas (Wed, 09 Jan 2019 19:39:08 GMT):
But yeah, absent a callback, this'll need probing.
yacovm (Wed, 09 Jan 2019 19:39:51 GMT):
so, we can either piggyback on the FSM tick
yacovm (Wed, 09 Jan 2019 19:40:03 GMT):
or make a new goroutine with a timer
kostas (Wed, 09 Jan 2019 19:40:42 GMT):
Piggybacking seems like a simpler approach?
yacovm (Wed, 09 Jan 2019 19:40:47 GMT):
When the sun shines again at where @guoger resides, he can think and say his opinion :)
yacovm (Wed, 09 Jan 2019 19:40:56 GMT):
I'm good with piggybacking
yacovm (Wed, 09 Jan 2019 19:41:12 GMT):
but then we won't have that 2nd chance / 2nd sweep you advocated for
kostas (Wed, 09 Jan 2019 19:41:26 GMT):
Why does everything in life have to be a tradeoff?
yacovm (Wed, 09 Jan 2019 19:41:31 GMT):
but i guess that in a way, loss of heartbeats from the leader takes that into account implictly
kostas (Wed, 09 Jan 2019 19:41:33 GMT):
A true bummer.
yacovm (Wed, 09 Jan 2019 19:42:16 GMT):
anyway i guess we can decide tomorrow when @guoger is awake
kostas (Wed, 09 Jan 2019 19:42:18 GMT):
> but i guess that in a way, loss of heartbeats from the leader takes that into account implictly
That makes sense and mitigates the original concern.
guoger (Thu, 10 Jan 2019 02:06:59 GMT):
I still don't get this... why couldn't `Errored` simply return a closed channel if node is candidate? The actually problem w.r.t
> I'm saying the following: Create a channel whenever we're not a candidate. Close that channel when we become a candidate. Return that channel via Errored. What am I missing?
is that, when the node becomes available, how to open the previously closed channel without contention (someone might be reading the channel while you change it). And I propose that every time `Errored` is called, it inspects current SoftState, and return a closed/open channel based on that. (a new channel being created and returned)
guoger (Thu, 10 Jan 2019 02:07:16 GMT):
cc @yacovm @kostas
kostas (Thu, 10 Jan 2019 02:55:33 GMT):
@guoger: That can work I think.
yacovm (Thu, 10 Jan 2019 08:27:09 GMT):
Returning an opened/closed channel won't suffice. Imagine you return an open channel, but then you detect the loss of leadership.
you need to close the open channel you returned, otherwise a connected peer won't know you lost leadership
yacovm (Thu, 10 Jan 2019 09:11:43 GMT):
What you did @guoger in the change set, seems right to me
yacovm (Thu, 10 Jan 2019 09:12:22 GMT):
but - are we guaranteed to get a notification via the `Ready` channel about a leadership loss, and a leadership election?
yacovm (Thu, 10 Jan 2019 09:12:33 GMT):
I guess we are because we need to add it to the log, right?
guoger (Thu, 10 Jan 2019 09:13:27 GMT):
yes we are guaranteed to get notified about softstate change. but no we don't need to persist this to log
yacovm (Thu, 10 Jan 2019 09:13:48 GMT):
oh, i see
guoger (Thu, 10 Jan 2019 09:16:16 GMT):
you are fast! i haven't added you to reviewer list yet :P
guoger (Thu, 10 Jan 2019 09:18:07 GMT):
is there an integration test to verify https://jira.hyperledger.org/browse/FAB-13438 ?
yacovm (Thu, 10 Jan 2019 09:25:01 GMT):
not that i know of...
yacovm (Thu, 10 Jan 2019 09:25:11 GMT):
and it will be hard to do one
yacovm (Thu, 10 Jan 2019 09:25:21 GMT):
because you need from one end, to connect to Deliver to the orderer
yacovm (Thu, 10 Jan 2019 09:25:29 GMT):
then to make a quorum loss
yacovm (Thu, 10 Jan 2019 09:25:33 GMT):
and then to ensure the deliver is closed
yacovm (Thu, 10 Jan 2019 09:25:50 GMT):
I suggest we check that in unit tests
yacovm (Thu, 10 Jan 2019 09:25:59 GMT):
like you did
yacovm (Thu, 10 Jan 2019 09:51:54 GMT):
@guoger can you take a look at this integration test failure? https://jenkins.hyperledger.org/job/fabric-verify-integration-tests-x86_64/4558/consoleFull
(1)
`orderer4` decides there is a leader:
```
00:52:19 [e][OrdererOrg.orderer4] 2019-01-09 22:51:38.695 UTC [orderer.consensus.etcdraft] Step -> INFO 135 4 [term: 1] received a MsgHeartbeat message with higher term from 3 [term: 2] {"channel": "testchannel2", "node": 4}
00:52:19 [e][OrdererOrg.orderer4] 2019-01-09 22:51:38.695 UTC [orderer.consensus.etcdraft] becomeFollower -> INFO 136 4 became follower at term 2 {"channel": "testchannel2", "node": 4}
00:52:19 [e][OrdererOrg.orderer4] 2019-01-09 22:51:38.695 UTC [orderer.consensus.etcdraft] run -> INFO 137 raft.node: 4 elected leader 3 at term 2 {"channel": "testchannel2", "node": 4}
00:52:19 [e][OrdererOrg.orderer4] 2019-01-09 22:51:38.697 UTC [orderer.consensus.etcdraft] serveRaft -> INFO 138 Raft leader changed: 0 -> 3 {"channel": "testchannel2", "node": 4}
```
But at the same second, node 2 starts an election:
```
00:52:19 [e][OrdererOrg.orderer2] 2019-01-09 22:51:47.037 UTC [orderer.consensus.etcdraft] Step -> INFO 112 2 is starting a new election at term 2 {"channel": "testchannel2", "node": 2}
00:52:19 [e][OrdererOrg.orderer2] 2019-01-09 22:51:47.038 UTC [orderer.consensus.etcdraft] becomePreCandidate -> INFO 113 2 became pre-candidate at term 2 {"channel": "testchannel2",
```
What is happening here? :thinking_face:
(2)
Now, it sends orderer4 a transaction, and it is lost... orderer 4 should forward it to a leader. I guess the transaction is lost because there is a leader switch, but why does it happen?
guoger (Thu, 10 Jan 2019 10:11:58 GMT):
hmm, is it a deterministic failure? orderer1 also start election in the log. I guess 2&1 do not receive heartbeats. it does seem odd though. we see `Failed to send StepRequest to 4`, so orderer3 should be sending out heartbeats to all nodes. If it's flaky, i guess i'll need to add some printf and run it repeatedly in CI
yacovm (Thu, 10 Jan 2019 10:41:36 GMT):
ot
yacovm (Thu, 10 Jan 2019 10:41:41 GMT):
it's not deterministic
yacovm (Thu, 10 Jan 2019 10:42:07 GMT):
I don't understand
yacovm (Thu, 10 Jan 2019 10:44:52 GMT):
`Failed to send StepRequest to 4` is fine
yacovm (Thu, 10 Jan 2019 10:45:02 GMT):
it is a newly booted node
yacovm (Thu, 10 Jan 2019 10:45:36 GMT):
what I don't understand is - why node 2 is starting an election
yacovm (Thu, 10 Jan 2019 10:46:05 GMT):
how is node 3 elected as a leader in time `00:52:19` and also in time `00:52:19`, node 2 starts election?
yacovm (Thu, 10 Jan 2019 10:46:49 GMT):
@guoger
guoger (Thu, 10 Jan 2019 10:47:58 GMT):
that's not node 3 being elected at `00:52:19`. it is always leader. that log entry only indicates that orderer4 _just learnt_ this fact.
guoger (Thu, 10 Jan 2019 10:48:34 GMT):
_both_ node 2 and 1 start election actually.
guoger (Thu, 10 Jan 2019 10:49:29 GMT):
> Raft leader changed: 0 -> 3
this is log should be interpreted as: from this node's point view, raft leader changes from A to B
yacovm (Thu, 10 Jan 2019 10:50:08 GMT):
Yeah, my bad.
yacovm (Thu, 10 Jan 2019 10:50:25 GMT):
but - do you know why is node 2 or node 1 starting election?
yacovm (Thu, 10 Jan 2019 10:51:41 GMT):
I don't see any errors of communication from nodes 2, 1
guoger (Thu, 10 Jan 2019 10:51:44 GMT):
> I guess 2&1 do not receive heartbeats
As for reason, idk, and i'll need to insert some printf and run in CI repeatedly
yacovm (Thu, 10 Jan 2019 10:52:24 GMT):
hmm ok, thanks
guoger (Thu, 10 Jan 2019 10:52:30 GMT):
btw, doesn't [this](https://github.com/hyperledger/fabric/blob/1203bd18e2ca115bc100b57b0ebf766b68a992e1/orderer/common/cluster/replication.go#L210) cause problem, when the config block used to onboard node is at height 1?
guoger (Thu, 10 Jan 2019 10:52:30 GMT):
btw, doesn't [this](https://github.com/hyperledger/fabric/blob/1203bd18e2ca115bc100b57b0ebf766b68a992e1/orderer/common/cluster/replication.go#L210) cause problem, when the config block used to onboard node is at height 1? in which case genesis block is not pulled
yacovm (Thu, 10 Jan 2019 10:53:29 GMT):
do you mean the system channel or an application channel?
guoger (Thu, 10 Jan 2019 10:53:43 GMT):
syschannel
yacovm (Thu, 10 Jan 2019 10:54:04 GMT):
you're saying the system channel has a sequence of 1, or a height of 1?
guoger (Thu, 10 Jan 2019 10:54:19 GMT):
sequence
guoger (Thu, 10 Jan 2019 10:54:42 GMT):
height of 1 indicates genesis block only, right?
yacovm (Thu, 10 Jan 2019 10:54:47 GMT):
right
yacovm (Thu, 10 Jan 2019 10:55:29 GMT):
I know this looks weird, and I also explicitly made my tests to not handle a case where the system channel's last config block sequence is 1
guoger (Thu, 10 Jan 2019 10:55:32 GMT):
if you start a network with genesis block (block 0), then add a node (this is block 1). And then you use block 1 to boot new node
guoger (Thu, 10 Jan 2019 10:55:32 GMT):
haha, so this is the easter egg?
yacovm (Thu, 10 Jan 2019 10:55:36 GMT):
because i find it unlikely
yacovm (Thu, 10 Jan 2019 10:55:55 GMT):
that you have have a last config block of 1
yacovm (Thu, 10 Jan 2019 10:56:01 GMT):
and you want to onboard a new orderer
yacovm (Thu, 10 Jan 2019 10:56:15 GMT):
because it means you have no application channels
yacovm (Thu, 10 Jan 2019 10:56:24 GMT):
and you suddenly decide to onboard a new orderer
yacovm (Thu, 10 Jan 2019 10:56:52 GMT):
usually when you setup a network, you create some channel
yacovm (Thu, 10 Jan 2019 10:57:01 GMT):
when you do that - the last config block's sequence will be 2 :)
guoger (Thu, 10 Jan 2019 10:57:25 GMT):
haha, so this is the easter egg?
yacovm (Thu, 10 Jan 2019 10:57:36 GMT):
eh, I wouldn't call it an easter egg
guoger (Thu, 10 Jan 2019 10:57:37 GMT):
(accidentally editted previous sentence...)
yacovm (Thu, 10 Jan 2019 10:57:49 GMT):
but i just find the use case far fetched
yacovm (Thu, 10 Jan 2019 10:58:17 GMT):
in any case you can "solve" this by making a new config transaction
guoger (Thu, 10 Jan 2019 10:58:24 GMT):
is it really hard to fix this? otherwise, we definitely need to document this
guoger (Thu, 10 Jan 2019 10:58:24 GMT):
is it really hard to fix this? otherwise, we definitely need to document this behavior
yacovm (Thu, 10 Jan 2019 10:59:10 GMT):
I'll open a JIRA for that and we'll prioritize this
guoger (Thu, 10 Jan 2019 10:59:54 GMT):
(i spent some time yesterday figuring this out when i was preparing for demo :joy: )
yacovm (Thu, 10 Jan 2019 12:30:55 GMT):
@guoger actually - if we have more than 2 blocks in the system channel... then latest height will be 2
yacovm (Thu, 10 Jan 2019 12:30:58 GMT):
so the loop should work...
yacovm (Thu, 10 Jan 2019 12:31:33 GMT):
```
func TestSystemChannelLatestSequenceIsOne(t *testing.T) {
systemChannelBlocks := createBlockChain(0, 1)
firstBlockResponse := &orderer.DeliverResponse{
Type: &orderer.DeliverResponse_Block{
Block: systemChannelBlocks[0],
},
}
secondBlockResponse := &orderer.DeliverResponse{
Type: &orderer.DeliverResponse_Block{
Block: systemChannelBlocks[1],
},
}
osn := newClusterNode(t)
defer osn.stop()
osn.blockResponses = make(chan *orderer.DeliverResponse, 10)
dialer := newCountingDialer()
bp := newBlockPuller(dialer, osn.srv.Address())
bp.FetchTimeout = time.Hour
bp.MaxPullBlockRetries = 1
channelLister := &mocks.ChannelLister{}
channelLister.On("Channels").Return([]cluster.ChannelGenesisBlock{})
channelLister.On("Close")
blocksCommittedToSystemLedger := make(chan *common.Block, 2)
systemLedger := &mocks.LedgerWriter{}
systemLedger.On("Append", mock.Anything).Return(nil).Run(func(arg mock.Arguments) {
blocksCommittedToSystemLedger <- arg.Get(0).(*common.Block)
})
systemLedger.On("Height").Return(func() uint64 {
return uint64(len(blocksCommittedToSystemLedger))
})
lf := &mocks.LedgerFactory{}
lf.On("Close")
lf.On("GetOrCreate", "system").Return(systemLedger, nil)
r := cluster.Replicator{
Filter: cluster.AnyChannel,
LedgerFactory: lf,
AmIPartOfChannel: func(configBlock *common.Block) error {
return cluster.ErrNotInChannel
},
Logger: flogging.MustGetLogger("test"),
SystemChannel: "system",
ChannelLister: channelLister,
Puller: bp,
BootBlock: systemChannelBlocks[1],
}
osn.addExpectProbeAssert()
osn.blockResponses <- secondBlockResponse
osn.addExpectProbeAssert()
osn.blockResponses <- secondBlockResponse
osn.addExpectPullAssert(0)
osn.blockResponses <- firstBlockResponse
osn.blockResponses <- secondBlockResponse
r.ReplicateChains()
bp.Close()
dialer.assertAllConnectionsClosed(t)
assert.Len(t, systemChannelBlocks, 2)
}
```
yacovm (Thu, 10 Jan 2019 12:31:52 GMT):
This unit test which I just wrote, replicates a system channel with a genesis block, and a config block on top of it
yacovm (Thu, 10 Jan 2019 12:32:19 GMT):
and it passes
guoger (Thu, 10 Jan 2019 12:47:39 GMT):
:thinking: tried again and you are right
guoger (Thu, 10 Jan 2019 16:07:37 GMT):
should we be concerned w.r.t integration test failure seen [here](https://jenkins.hyperledger.org/job/fabric-verify-integration-tests-x86_64/4600/console) and [here](https://jenkins.hyperledger.org/job/fabric-verify-integration-tests-x86_64/4603/console)?
```
23:30:25 Summarizing 1 Failure:
23:30:25
23:30:25 [Fail] PrivateData reconciliation [BeforeEach] verify private data reconciliation when adding a new org to collection config
```
```
23:25:51 Summarizing 2 Failures:
23:25:51
23:25:51 [Fail] EndToEndACL [It] enforces access control list policies
23:25:51 /w/workspace/fabric-verify-integration-tests-x86_64/gopath/src/github.com/hyperledger/fabric/integration/e2e/acl_test.go:181
23:25:51
23:25:51 [Fail] EndToEnd basic kafka network with 2 orgs [BeforeEach] executes a basic kafka network with 2 orgs
23:25:51 /w/workspace/fabric-verify-integration-tests-x86_64/gopath/src/github.com/hyperledger/fabric/integration/e2e/e2e_test.go:135
```
yacovm (Thu, 10 Jan 2019 16:15:57 GMT):
@sykesm do you have the above in your collection? :thinking_face:
sykesm (Thu, 10 Jan 2019 16:19:56 GMT):
the collection is here: https://jira.hyperledger.org/issues/?filter=12178
yacovm (Thu, 10 Jan 2019 16:20:11 GMT):
thanks, I'll look :)
sykesm (Thu, 10 Jan 2019 16:20:27 GMT):
Looks like FAB-13594
yacovm (Thu, 10 Jan 2019 16:21:33 GMT):
but FAB-13594 doesn't have 181 in the stack trace
sykesm (Thu, 10 Jan 2019 16:22:19 GMT):
don't they both call the same helper function for updating the config block?
yacovm (Thu, 10 Jan 2019 16:24:13 GMT):
```
17:11:25 Saw:
17:11:25 Get instantiated chaincodes on channel testchannel:
17:11:25 Name: mycc, Version: 0.0, Path: github.com/hyperledger/fabric/integration/chaincode/simple/cmd, Escc: escc, Vscc: vscc
17:11:25
17:11:25 Which matches the unexpected:
17:11:25 Name: mycc, Version: 0.0, Path: .*, Escc: escc, Vscc: vscc
```
yacovm (Thu, 10 Jan 2019 16:24:25 GMT):
this is the error in the ACL test that Jay linked
yacovm (Thu, 10 Jan 2019 16:24:45 GMT):
while in FAB-13594 the error is that you get an unexpected status
yacovm (Thu, 10 Jan 2019 16:24:45 GMT):
while in FAB-13594 the error is that you get an unexpected status - `can't read the block: &{FORBIDDEN}`
yacovm (Thu, 10 Jan 2019 16:31:07 GMT):
@sykesm
yacovm (Fri, 11 Jan 2019 20:43:53 GMT):
I added an [integration test ](https://gerrit.hyperledger.org/r/#/c/28648/) that find the current Raft leader, kills it, and makes sure a new one is elected. It only runs in 20 seconds, and I think it's worth having.
kostas (Fri, 11 Jan 2019 21:03:27 GMT):
Had a look, also think it's worth having. Thanks for putting this together.
guoger (Sat, 12 Jan 2019 03:21:05 GMT):
@yacovm but how's that different from the one we already had in that file? (or at least you could just augment that test to perform leader checks, which is probably still redundant IMO, cuz if no leader exists, that test couldn't perform tx)
guoger (Sat, 12 Jan 2019 03:21:26 GMT):
one more test doesn't harm, just checking the dealt
guoger (Sat, 12 Jan 2019 03:21:26 GMT):
one more test doesn't harm, just checking the delta
guoger (Sat, 12 Jan 2019 03:30:09 GMT):
and for our fake clocks, i opened an issue [there](https://github.com/cloudfoundry/clock/issues/6). Welcome some work around ideas for now (this is the main factor that slows the test down) cc @sykesm
guoger (Sat, 12 Jan 2019 03:30:09 GMT):
and for our fake clocks, i opened an issue [there](https://github.com/cloudfoundry/clock/issues/6). Welcome some work around ideas for now (this is the main factor that slows the etcdraft test down) cc @sykesm
guoger (Sat, 12 Jan 2019 03:38:12 GMT):
and a question (may be very dumb) to @tock , when migrate to raft, one should submit `configUpdate` that changes ConsensusType from `kafka` to `etcdraft` *only* to system channel. And system channel will cascade that to app channels, am i getting this right?
guoger (Sat, 12 Jan 2019 03:38:12 GMT):
and a question (may be very dumb) to @tock , when migrate to raft, one should submit `configUpdate` that changes ConsensusType from `kafka` to `etcdraft` *ONLY* to system channel. And system channel will cascade that to app channels, am i getting this right?
guoger (Sat, 12 Jan 2019 03:38:12 GMT):
and a question (may be very dumb) to @tock , when migrate to raft, one should submit `configUpdate` that changes ConsensusType from `kafka` to `etcdraft` *ONLY* to system channel. ~And system channel will cascade that to app channels, am i getting this right?~ from the description of FAB-13264, i figured this is not the case... but why should user do it for app channels?
guoger (Sat, 12 Jan 2019 04:01:16 GMT):
also, could you briefly summarize the use of `ConsensusMigrationContext` for sys-channel and app-channel, if this is not too much to ask. it'll help me understanding the code while doing review. thx!
guoger (Sat, 12 Jan 2019 04:01:16 GMT):
also, could you briefly summarize the use of `ConsensusMigrationContext` for sys-channel and app-channel, if this is not too much to ask. it'll help me understanding the code while doing review (mostly because i couldn't find it in design doc). thx!
guoger (Sat, 12 Jan 2019 04:08:50 GMT):
in design doc:
> If anything goes wrong after BEGIN-MIG, one may invoke ABORT-MIG to abort the migration.
what's failure scenario we are talking about here? can user abort during migration?
guoger (Sat, 12 Jan 2019 04:10:00 GMT):
(pls don't take me wrong, i'm not jumping out in the middle of development and criticize the design. I'm simply trying to understand this...)
yacovm (Sat, 12 Jan 2019 07:07:21 GMT):
> but how's that different from the one we already had in that file?
because you don't find who is the leader and kill it. you just kill some orderer who may not be the leader, no? @guoger
guoger (Sat, 12 Jan 2019 07:08:47 GMT):
ahh, true, thx!
yacovm (Sat, 12 Jan 2019 14:59:27 GMT):
can anyone look at [this](https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/7883/console) ?
yacovm (Sat, 12 Jan 2019 15:00:45 GMT):
I see here 2 failures:
1) A test of mine failed:
```
16:35:21 --- FAIL: TestTicker (0.08s)
16:35:21 --- FAIL: TestTicker/Stop_ticker_serially (0.02s)
16:35:21
yacovm (Sat, 12 Jan 2019 15:02:37 GMT):
I opened [FAB-13653](https://jira.hyperledger.org/browse/FAB-13653) for mine, but does anyone has a clue what's the other one?
guoger (Sat, 12 Jan 2019 15:09:14 GMT):
is it deterministic?
guoger (Sat, 12 Jan 2019 15:09:29 GMT):
btw, how to convert a story to bug? i accidentally created one with wrong type....
yacovm (Sat, 12 Jan 2019 15:11:31 GMT):
it's in CI, so of course it's not
yacovm (Sat, 12 Jan 2019 15:11:54 GMT):
which FAB?
guoger (Sat, 12 Jan 2019 15:12:52 GMT):
https://jira.hyperledger.org/browse/FAB-13652
yacovm (Sat, 12 Jan 2019 15:13:52 GMT):
ah you just click F you click F5 on your keyboard
yacovm (Sat, 12 Jan 2019 15:14:08 GMT):
ah you just click F5 on your keyboard
guoger (Sat, 12 Jan 2019 15:14:30 GMT):
lol
guoger (Sat, 12 Jan 2019 15:15:40 GMT):
but seriously, how did you do that? i don't see `type` in edit tab
yacovm (Sat, 12 Jan 2019 15:16:29 GMT):
Answered in PM
kostas (Sat, 12 Jan 2019 17:32:12 GMT):
Oh man, worst timing ever.
kostas (Sat, 12 Jan 2019 17:33:56 GMT):
So, the more Raft groups the more overhead you have to deal with (in terms of requests that need not be duplicated --or can be batched better-- such as heartbeats, etc.).
kostas (Sat, 12 Jan 2019 17:33:56 GMT):
So, the more Raft groups you have, the bigger the overhead an OSN has to deal with (in terms of requests that need not be duplicated --or can be batched better-- such as heartbeats, etc.).
kostas (Sat, 12 Jan 2019 17:34:29 GMT):
Remember the MultiRaft effort by CockroachDB that was abandoned: https://www.cockroachlabs.com/blog/scaling-raft/ (I had linked to that in the design doc)
kostas (Sat, 12 Jan 2019 17:34:45 GMT):
Today I found this: https://github.com/lni/dragonboat
kostas (Sat, 12 Jan 2019 17:36:00 GMT):
(First commit 9 days ago, 1.2K stars on GitHub already :astonished:)
yacovm (Sat, 12 Jan 2019 18:03:43 GMT):
But we manage everything in Fabric via per channel configuration
yacovm (Sat, 12 Jan 2019 18:04:39 GMT):
if we have a Raft FSM serving several channels then we run into existential and philosophical questions of how to map the configuration to the real world
yacovm (Sat, 12 Jan 2019 18:05:03 GMT):
I, for one - like the one-to-one and onto mapping of the channels and Raft instances
guoger (Sat, 12 Jan 2019 22:40:12 GMT):
I actually happened to read a bit about dragonboat today. Most of optimization there is on IO and batching, as well as key value storage
guoger (Sat, 12 Jan 2019 22:40:55 GMT):
Do you see any particular piece we could take as reference there? @kostas
guoger (Sat, 12 Jan 2019 22:41:35 GMT):
But this is true... @kostas [ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=oRjgNC8BXBCuuzT8i)
guoger (Sat, 12 Jan 2019 23:12:59 GMT):
One thing we could potentially do to optimize the heartbeats, is to creat our own layer of connectivity check, and intercept heartbeat messages
guoger (Sat, 12 Jan 2019 23:16:21 GMT):
The layer simply does a full mesh connectivity check. Whenever a node sends out a heartbeat, we lookup the connectivity, and feed it with artificial MsgHeartbeatResp
guoger (Sat, 12 Jan 2019 23:16:21 GMT):
The layer simply does a full mesh connectivity check. Whenever a node sends out a heartbeat, we look up the connectivity status, and feed it with artificial MsgHeartbeatResp if the connection between src and dest has been active recently (<1s)
guoger (Sat, 12 Jan 2019 23:16:57 GMT):
And we periodically give followers MsgHeartbeatResp based on connectivity as well
yacovm (Sun, 13 Jan 2019 07:13:51 GMT):
why not just make the heart beat interval bigger @guoger ?
yacovm (Sun, 13 Jan 2019 07:15:37 GMT):
> One thing we could potentially do to optimize the heartbeats, is to creat our own layer of connectivity check, and intercept heartbeat messages
that won't solve the I/O problem because it also writes these heart beats into the disk, doesn't it?
yacovm (Sun, 13 Jan 2019 07:16:39 GMT):
I say we first finish Raft and make it solid and sturdy and only later think how we make it fast for multi-tenancy.
guoger (Sun, 13 Jan 2019 07:20:56 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=RtH7Gas7ZWdci9YXB) @yacovm that's probably the best option we have for now, although it prolongs leader failover time a bit
guoger (Sun, 13 Jan 2019 07:21:32 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=zSM9bE5kZBspvQGNh) @yacovm it's not trying to solve i/o problem. it's just a means to reduce number of heartbeat messages. plus, heartbeat is not written to disk
yacovm (Sun, 13 Jan 2019 07:21:43 GMT):
that's good
guoger (Sun, 13 Jan 2019 07:22:06 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=KiqrnF4t7TT6SDXND) @yacovm i agree. just trying to brainstorm some ideas here :)
yacovm (Sun, 13 Jan 2019 07:22:17 GMT):
look at the upside, at least we have a single gRPC connection between nodes regardless how many channels
yacovm (Sun, 13 Jan 2019 07:22:37 GMT):
well, actually 2 - egress and ingress
yacovm (Sun, 13 Jan 2019 07:22:44 GMT):
gossip has a single...
guoger (Sun, 13 Jan 2019 07:25:30 GMT):
another problem i often see is:
> 22:12:40 [e][OrdererOrg.orderer1] 2019-01-12 14:11:31.545331 W | wal: sync duration of 1.438320949s, expected less than 1s
guoger (Sun, 13 Jan 2019 07:26:54 GMT):
this is on critical path where data is persisted to wal.
guoger (Sun, 13 Jan 2019 07:29:30 GMT):
and 1.4s looks pretty bad... it blocks raft thread, which prevents next round of heartbeat from being broadcasted.
yacovm (Sun, 13 Jan 2019 07:29:34 GMT):
where do you see it?
guoger (Sun, 13 Jan 2019 07:29:42 GMT):
only in CI
yacovm (Sun, 13 Jan 2019 07:29:47 GMT):
so no problem
yacovm (Sun, 13 Jan 2019 07:30:06 GMT):
we can document in the documentation site - not run Raft in production on CI servers
yacovm (Sun, 13 Jan 2019 07:30:06 GMT):
we can document in the documentation site - not run Raft in production on our CI servers
guoger (Sun, 13 Jan 2019 07:30:16 GMT):
lol
guoger (Sun, 13 Jan 2019 07:30:34 GMT):
actually, have you ever seen this in svt tests?
yacovm (Sun, 13 Jan 2019 07:30:42 GMT):
i have SSD, and no
yacovm (Sun, 13 Jan 2019 07:30:42 GMT):
no
yacovm (Sun, 13 Jan 2019 07:31:16 GMT):
I think we really should relax the times though
yacovm (Sun, 13 Jan 2019 07:31:30 GMT):
100 milliseconds for a heart beat sounds a bit edgy to me
yacovm (Sun, 13 Jan 2019 07:31:36 GMT):
what are the recommended values?
guoger (Sun, 13 Jan 2019 07:31:48 GMT):
that's the default value for etcd
yacovm (Sun, 13 Jan 2019 07:32:03 GMT):
what are the recommended values for production?
yacovm (Sun, 13 Jan 2019 07:32:16 GMT):
what does etcd use?
guoger (Sun, 13 Jan 2019 07:32:33 GMT):
and from https://coreos.com/etcd/docs/latest/tuning.html:
> The value of heartbeat interval is recommended to be around the maximum of average round-trip time (RTT) between members, normally around 0.5-1.5x the round-trip time.
yacovm (Sun, 13 Jan 2019 07:35:42 GMT):
by the way now that you mention it
yacovm (Sun, 13 Jan 2019 07:35:51 GMT):
do we support dynamic changing of this value? :thinking:
guoger (Sun, 13 Jan 2019 08:35:12 GMT):
good point. no, and we should guard against this...
guoger (Sun, 13 Jan 2019 08:36:18 GMT):
i don't see a good reason to make this changeable, at least for now...
guoger (Sun, 13 Jan 2019 08:37:15 GMT):
although, we probably should make `SnapshotInterval` changeable, and as you suggested before, per size instead of # of blocks. although i think that's not at top priority
yacovm (Sun, 13 Jan 2019 08:42:54 GMT):
we should document all these easter eggs in JIRA :/
yacovm (Sun, 13 Jan 2019 08:42:59 GMT):
to not lose them
yacovm (Sun, 13 Jan 2019 08:43:11 GMT):
or are they already documented?
guoger (Sun, 13 Jan 2019 08:56:45 GMT):
good point, created FAB-13654 FAB-13655 FAB-13656
guoger (Mon, 14 Jan 2019 10:08:13 GMT):
need a another pair of eyes for [this](https://jenkins.hyperledger.org/job/fabric-verify-integration-tests-x86_64/4704/consoleFull)
seems that `Inovke` is successful and peer committed the block,
```
14:42:01 [e][Org2.peer0] 2019-01-14 06:41:59.561 UTC [kvledger] CommitWithPvtData -> INFO 057 [testchannel] Committed block [8] with 1 transaction(s) in 28ms (state_validation=0ms block_commit=19ms state_commit=4ms)
14:42:01 [e][Org1.peer0] 2019-01-14 06:41:59.561 UTC [kvledger] CommitWithPvtData -> INFO 08a [testchannel] Committed block [8] with 1 transaction(s) in 27ms (state_validation=0ms block_commit=18ms state_commit=4ms)
14:42:01 [e][peer-chaincode-invoke] 2019-01-14 06:41:59.561 UTC [chaincodeCmd] ClientWait -> INFO 001 txid [86e9b9034cf6a469cdbec286d2999cb306c4024ccd3d6af42686ff6b843518a2] committed with status (VALID) at 127.0.0.1:33009
14:42:01 [e][Org2.peer1] 2019-01-14 06:41:59.563 UTC [kvledger] CommitWithPvtData -> INFO 082 [testchannel] Committed block [8] with 1 transaction(s) in 27ms (state_validation=0ms block_commit=18ms state_commit=4ms)
14:42:01 [e][peer-chaincode-invoke] 2019-01-14 06:41:59.563 UTC [chaincodeCmd] ClientWait -> INFO 002 txid [86e9b9034cf6a469cdbec286d2999cb306c4024ccd3d6af42686ff6b843518a2] committed with status (VALID) at 127.0.0.1:33024
14:42:01 [e][peer-chaincode-invoke] 2019-01-14 06:41:59.563 UTC [chaincodeCmd] chaincodeInvokeOrQuery -> INFO 003 Chaincode invoke successful. result: status:200
```
However, `Query` failed with a stale value...
```
14:42:01 [e][Org1.peer1] 2019-01-14 06:41:59.635 UTC [endorser] callChaincode -> INFO 077 [testchannel][4b55e50b] Entry chaincode: name:"mycc"
14:42:01 [e][Org1.peer1] 2019-01-14 06:41:59.635 UTC [peer.chaincode.6nacn2bmlfenfpvac6ca5uumbi-Org1.peer1-mycc-0.0] func2 -> INFO 078 ex02 Invoke
14:42:01 [e][Org1.peer1] 2019-01-14 06:41:59.636 UTC [peer.chaincode.6nacn2bmlfenfpvac6ca5uumbi-Org1.peer1-mycc-0.0] func2 -> INFO 079 Query Response:{"Name":"a","Amount":"40"}
```
guoger (Mon, 14 Jan 2019 13:42:57 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=KANgekNWAsBRbPWwe) @tock here :) and also state machine diagram would be super helpful
kostas (Mon, 14 Jan 2019 13:45:45 GMT):
> Do you see any particular piece we could take as reference there? @kostas
RE: Dragonboat. Haven't studied the codebase in detail yet, just a quick skim, so no. But as we wrote before, we are certainly exposed to the same issues for which the Multiraft project was started. At any rate, I agree w/ you guys. Nothing to sweat about at this point. Just posted it so that it's out there. We can optimize later if need be.
tock (Mon, 14 Jan 2019 13:46:25 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=NWm4xR5JzjmJobz3t) The user will have to put a config-update with state=CONTEXT, type=etcdraft, and rarft-metadata on every app channel. The reason is, that once a channel is created, the system channel really has no way to enforce anything on it ("cascade"). In order to have a valid etcdraft config-block on an app channel, it has to come from a config-update on that channel.
kostas (Mon, 14 Jan 2019 13:47:10 GMT):
RE: RPC changes. +1 on calling everything `Step` as the etcd/raft library does, similar to what Jay suggested in the scrum. So, one `Step` stream to rule everything.
yacovm (Mon, 14 Jan 2019 13:52:59 GMT):
ok, so i'll just add a field `Type` to the protobuf message we send that would denote the recipient of the message - i.e, a transaction or a consensus message
yacovm (Mon, 14 Jan 2019 13:53:17 GMT):
sounds good, @kostas @guoger ?
yacovm (Mon, 14 Jan 2019 13:54:28 GMT):
ok we have a quorum
tock (Mon, 14 Jan 2019 17:07:38 GMT):
@kostas @yacovm @C0rWin @guoger I split the migration green path into 4 parts:
https://gerrit.hyperledger.org/r/#/c/28435/
https://gerrit.hyperledger.org/r/#/c/28705/
https://gerrit.hyperledger.org/r/#/c/28707/
https://gerrit.hyperledger.org/r/#/c/28708/
jyellick (Tue, 15 Jan 2019 04:26:10 GMT):
This doesn't seem to match any of the existing test flakes: https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/7974/console but wanted to confirm before I opened a bug
guoger (Tue, 15 Jan 2019 05:36:33 GMT):
@jyellick thanks for reporting this. i haven't seen this either, pls open a bug under https://jira.hyperledger.org/browse/FAB-13371, thx!
jyellick (Tue, 15 Jan 2019 06:23:26 GMT):
https://jira.hyperledger.org/browse/FAB-13691
tock (Tue, 15 Jan 2019 08:21:09 GMT):
@guoger is this something you are familiar with? etcdraft failing integration, https://jenkins.hyperledger.org/job/fabric-verify-integration-tests-x86_64/4718/console
guoger (Tue, 15 Jan 2019 08:30:21 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=MgzAN4k5m3MSqpMpG) @yacovm posted this before. It hasn't hit etcdraft chain yet, but i do find this failure weird... could you open a jira for this, if yacov hasn't done it.
yacovm (Tue, 15 Jan 2019 08:30:54 GMT):
I didn't open a JIRA for the 2nd one
yacovm (Tue, 15 Jan 2019 08:31:11 GMT):
that's the reason for the failure I think
guoger (Tue, 15 Jan 2019 08:45:30 GMT):
what's the reason?
yacovm (Tue, 15 Jan 2019 08:53:59 GMT):
well if you have an envelope that is swallowed
yacovm (Tue, 15 Jan 2019 08:54:03 GMT):
the test will fail....
yacovm (Tue, 15 Jan 2019 08:54:22 GMT):
the broadcast client causes a panic which causes the panic handler to swallow the envelope
yacovm (Tue, 15 Jan 2019 08:54:28 GMT):
well, more like puke it
yacovm (Tue, 15 Jan 2019 08:54:31 GMT):
instead of swallowing
tock (Tue, 15 Jan 2019 09:22:38 GMT):
@yacovm @guoger is this related to the previous one? it looks different to my untrained eye: https://jenkins.hyperledger.org/job/fabric-verify-integration-tests-x86_64/4718/console
tock (Tue, 15 Jan 2019 09:22:38 GMT):
@yacovm @guoger is this related to the previous one? it looks different to my untrained eye (it is a rerun): https://jenkins.hyperledger.org/job/fabric-verify-integration-tests-x86_64/4718/console
guoger (Tue, 15 Jan 2019 10:32:13 GMT):
hmmm... it looks like the same panic stack?
guoger (Tue, 15 Jan 2019 10:32:45 GMT):
i'm probably missing something... wdyt @yacovm
yacovm (Tue, 15 Jan 2019 10:34:37 GMT):
i agree
jlcs (Tue, 15 Jan 2019 10:52:06 GMT):
Has joined the channel.
tock (Tue, 15 Jan 2019 13:06:18 GMT):
@guoger @yacovm my bad... I split the CR and left a hole in the etcdraft/chain... sorry for wasting your time :flushed:
yacovm (Tue, 15 Jan 2019 13:13:41 GMT):
a hole in the chain
yacovm (Tue, 15 Jan 2019 13:13:49 GMT):
sounds painful
guoger (Wed, 16 Jan 2019 02:38:03 GMT):
@tock nothing to be sorry for :) we observed this failure before anyway. Btw, do you have the state machine diagram handy? thx
guoger (Wed, 16 Jan 2019 03:35:21 GMT):
@tock i'm confused... can `migration_context` be > 0 on system channel? in proto, it states:
> // On a standard channel, this must be >0. On a system channel, this is =0.
although in the code, it checks:
```
sysState, sysContext := r.systemChannel.MigrationStatus().StateContext()
if !(sysState == ab.ConsensusType_MIG_STATE_START && sysContext > 0) {
err = fmt.Errorf("Cannot commit consensus-type migration because system channel (%s): state=%s, context=%d (expect: state=%s, context>0)",
r.systemChannel.ChainID(), sysState.String(), sysContext, ab.ConsensusType_MIG_STATE_START)
return err
}
```
tock (Wed, 16 Jan 2019 07:35:59 GMT):
correct. the documentation in protos needs updating. I decided to enforce the relation between start and commit+abort with the context in the system channel too. good catch, I'll update it. thanks!
tock (Wed, 16 Jan 2019 07:35:59 GMT):
@guoger correct. the documentation in protos needs updating. I decided to enforce the relation between start and commit+abort with the context in the system channel too. good catch, I'll update it. thanks!
I created a JIRA so that I won't forget: https://jira.hyperledger.org/browse/FAB-13704
tock (Wed, 16 Jan 2019 08:04:09 GMT):
@jyellick I have a question about `Bundle.ValidateNew()` and `ConsortiumsConfig()`:
Looking at the code in `multichannel.Registrar`, I understood that `_, ok := ConsortiumsConfig()` is used to identify the system channel. If two system channels are identified, the code panics.
Now, in `Bundle.ValidateNew()`, we have:
```golang
if cc, ok := b.ConsortiumsConfig(); ok {
ncc, ok := nb.ConsortiumsConfig()
if !ok {
return errors.Errorf("Current config has consortiums section, but new config does not")
}
for consortiumName, consortium := range cc.Consortiums() {
nconsortium, ok := ncc.Consortiums()[consortiumName]
if !ok {
continue
}
for orgName, org := range consortium.Organizations() {
norg, ok := nconsortium.Organizations()[orgName]
if !ok {
continue
}
mspID := org.MSPID()
if mspID != norg.MSPID() {
return errors.Errorf("Consortium %s org %s attempted to change MSP ID from %s to %s", consortiumName, orgName, mspID, norg.MSPID())
}
}
}
}
```
what prevents a new config update on a standard channel from adding a `ConsortiumsConfig` ?
tock (Wed, 16 Jan 2019 08:04:09 GMT):
@jyellick I have a question about `Bundle.ValidateNew()` and `ConsortiumsConfig()`:
Looking at the code in `multichannel.Registrar`, I understood that `_, ok := ConsortiumsConfig()` is used to identify the system channel. If two system channels are identified, the code panics.
Now, in `Bundle.ValidateNew()`, we have:
```go
if cc, ok := b.ConsortiumsConfig(); ok {
ncc, ok := nb.ConsortiumsConfig()
if !ok {
return errors.Errorf("Current config has consortiums section, but new config does not")
}
for consortiumName, consortium := range cc.Consortiums() {
nconsortium, ok := ncc.Consortiums()[consortiumName]
if !ok {
continue
}
for orgName, org := range consortium.Organizations() {
norg, ok := nconsortium.Organizations()[orgName]
if !ok {
continue
}
mspID := org.MSPID()
if mspID != norg.MSPID() {
return errors.Errorf("Consortium %s org %s attempted to change MSP ID from %s to %s", consortiumName, orgName, mspID, norg.MSPID())
}
}
}
}
```
what prevents a new config update on a standard channel from adding a `ConsortiumsConfig` ?
tock (Wed, 16 Jan 2019 08:04:09 GMT):
@jyellick I have a question about `Bundle.ValidateNew()` and `ConsortiumsConfig()`:
Looking at the code in `multichannel.Registrar`, I understood that `_, ok := ConsortiumsConfig()` is used to identify the system channel. If two system channels are identified, the code panics.
Now, in `Bundle.ValidateNew()`, we have:
```
if cc, ok := b.ConsortiumsConfig(); ok {
ncc, ok := nb.ConsortiumsConfig()
if !ok {
return errors.Errorf("Current config has consortiums section, but new config does not")
}
for consortiumName, consortium := range cc.Consortiums() {
nconsortium, ok := ncc.Consortiums()[consortiumName]
if !ok {
continue
}
for orgName, org := range consortium.Organizations() {
norg, ok := nconsortium.Organizations()[orgName]
if !ok {
continue
}
mspID := org.MSPID()
if mspID != norg.MSPID() {
return errors.Errorf("Consortium %s org %s attempted to change MSP ID from %s to %s", consortiumName, orgName, mspID, norg.MSPID())
}
}
}
}
```
what prevents a new config update on a standard channel from adding a `ConsortiumsConfig` ?
tock (Wed, 16 Jan 2019 08:04:09 GMT):
@jyellick I have a question about `Bundle.ValidateNew()` and `ConsortiumsConfig()`:
Looking at the code in `multichannel.Registrar`, I understood that `_, ok := ConsortiumsConfig()` is used to identify the system channel. If two system channels are identified, the code panics.
Now, in `Bundle.ValidateNew()`, we have:
```
```
what prevents a new config update on a standard channel from adding a `ConsortiumsConfig` ?
tock (Wed, 16 Jan 2019 08:04:09 GMT):
@jyellick I have a question about `Bundle.ValidateNew()` and `ConsortiumsConfig()`:
Looking at the code in `multichannel.Registrar`, I understood that `_, ok := ConsortiumsConfig()` is used to identify the system channel. If two system channels are identified, the code panics.
Now, in `Bundle.ValidateNew()`, we have:
```
if cc, ok := b.ConsortiumsConfig(); ok {
ncc, ok := nb.ConsortiumsConfig()
if !ok {
return errors.Errorf("Current config has consortiums section, but new config does not")
}
for consortiumName, consortium := range cc.Consortiums() {
nconsortium, ok := ncc.Consortiums()[consortiumName]
if !ok {
continue
}
for orgName, org := range consortium.Organizations() {
norg, ok := nconsortium.Organizations()[orgName]
if !ok {
continue
}
mspID := org.MSPID()
if mspID != norg.MSPID() {
return errors.Errorf("Consortium %s org %s attempted to change MSP ID from %s to %s", consortiumName, orgName, mspID, norg.MSPID())
}
}
}
}
```
what prevents a new config update on a standard channel from adding a `ConsortiumsConfig` ?
tock (Wed, 16 Jan 2019 08:04:09 GMT):
@jyellick I have a question about `Bundle.ValidateNew()` and `ConsortiumsConfig()`:
Looking at the code in `multichannel.Registrar`, I understood that `_, ok := ConsortiumsConfig()` is used to identify the system channel. If two system channels are identified, the code panics.
Now, in `Bundle.ValidateNew()`, we have:
```
if cc, ok := b.ConsortiumsConfig(); ok {
ncc, ok := nb.ConsortiumsConfig()
if !ok {
return errors.Errorf("Current config has consortiums section, but new config does not")
}
for consortiumName, consortium := range cc.Consortiums() {
nconsortium, ok := ncc.Consortiums()[consortiumName]
if !ok {
continue
}
for orgName, org := range consortium.Organizations() {
norg, ok := nconsortium.Organizations()[orgName]
if !ok {
continue
}
mspID := org.MSPID()
if mspID != norg.MSPID() {
return errors.Errorf("Consortium %s org %s attempted to change MSP ID from %s to %s", consortiumName, orgName, mspID, norg.MSPID())
}
}
}
}
```
What prevents a new config update on a standard channel from adding a `ConsortiumsConfig` ?
If it is allowed to happen, this could mess up the next Registrar initialization. Am I missing something?
tock (Wed, 16 Jan 2019 08:04:09 GMT):
@jyellick I have a question about `Bundle.ValidateNew()` and `ConsortiumsConfig()`:
Looking at the code in `multichannel.Registrar`, I understood that `_, ok := ledgerResources.ConsortiumsConfig()` is used to identify the system channel. If two system channels are identified, the code panics.
Now, in `Bundle.ValidateNew()`, we have:
```
if cc, ok := b.ConsortiumsConfig(); ok {
ncc, ok := nb.ConsortiumsConfig()
if !ok {
return errors.Errorf("Current config has consortiums section, but new config does not")
}
for consortiumName, consortium := range cc.Consortiums() {
nconsortium, ok := ncc.Consortiums()[consortiumName]
if !ok {
continue
}
for orgName, org := range consortium.Organizations() {
norg, ok := nconsortium.Organizations()[orgName]
if !ok {
continue
}
mspID := org.MSPID()
if mspID != norg.MSPID() {
return errors.Errorf("Consortium %s org %s attempted to change MSP ID from %s to %s", consortiumName, orgName, mspID, norg.MSPID())
}
}
}
}
```
What prevents a new config update on a standard channel from adding a `ConsortiumsConfig` ?
If it is allowed to happen, this could mess up the next Registrar initialization. Am I missing something?
sanket1211 (Wed, 16 Jan 2019 11:23:11 GMT):
Has left the channel.
tock (Wed, 16 Jan 2019 12:23:21 GMT):
@guoger @yacovm @C0rWin @kostas Here is a draft document that describes the migration state machine: https://docs.google.com/document/d/1g_M8KO9cdbYMLBsscHd6ffv-X_ZRLeHvDcIEoV7YXUE/edit?usp=sharing
guoger (Wed, 16 Jan 2019 13:19:44 GMT):
@tock can you make it commentable?
tock (Wed, 16 Jan 2019 13:51:49 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=D6XidjSzKmfLxa4FD) @guoger https://docs.google.com/document/d/1g_M8KO9cdbYMLBsscHd6ffv-X_ZRLeHvDcIEoV7YXUE/edit?usp=sharing
kostas (Wed, 16 Jan 2019 15:37:14 GMT):
> What prevents a new config update on a standard channel from adding a `ConsortiumsConfig` ?
> If it is allowed to happen, this could mess up the next Registrar initialization. Am I missing something?
kostas (Wed, 16 Jan 2019 15:37:38 GMT):
@tock: I am not jyellick, but you're reading this right and not missing anything.
kostas (Wed, 16 Jan 2019 15:38:11 GMT):
(It will be hilarious if I'm wrong though.)
yacovm (Wed, 16 Jan 2019 15:39:04 GMT):
luckily, channel config transactions are too complex to be a good way to shoot yourself in the foot
jyellick (Wed, 16 Jan 2019 15:39:13 GMT):
Yes, the only thing protecting the network from getting messed up with initialization is that the orderer admin should not agree to define Consortiums on an application channel.
tock (Wed, 16 Jan 2019 16:10:23 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=LXFdBCNRr7TW8Y6Hn) @jyellick Do you object if I fix this while I am doing the migration work? I am not sure all orderer admins realize the implications of this...
tock (Wed, 16 Jan 2019 16:12:17 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=vyc4vQQaebujXqbhW) @yacovm Security by obscurity ;-)
jyellick (Wed, 16 Jan 2019 16:15:35 GMT):
@tock By all means. https://github.com/hyperledger/fabric/blob/f80f06953cf8eca38c9d428533f2547a953d983a/common/channelconfig/bundle.go#L79-L148
You will need to modify this function to allow migration from one consensus type to another. This is also where the check for addition of a consortiums group should go. Both should likely be protected by a capability to avoid forking.
tock (Wed, 16 Jan 2019 16:22:56 GMT):
Good. I'll do it in the context of this task: https://jira.hyperledger.org/browse/FAB-13705
tock (Thu, 17 Jan 2019 12:21:28 GMT):
@yacovm @kostas @C0rWin
I added these two to the bottom of the stack because they are easy and improve upon code that was already merged:
https://gerrit.hyperledger.org/r/#/c/28776/ - this one is just documentation https://jira.hyperledger.org/browse/FAB-13704 following @guoger 's comments
https://gerrit.hyperledger.org/r/#/c/28780/ - @jyellick this one addresses the issue with ConsortiumsConfig discussed above: https://jira.hyperledger.org/browse/FAB-13705
kostas (Thu, 17 Jan 2019 15:33:30 GMT):
Do we know what's causing the verify build failures in CI?
guoger (Thu, 17 Jan 2019 15:39:26 GMT):
@kostas which one?
kostas (Thu, 17 Jan 2019 15:44:07 GMT):
I was looking at https://gerrit.hyperledger.org/r/c/27845/ but I'm asking more generally. Why on earth does verify-build fail so often?
yacovm (Thu, 17 Jan 2019 15:47:48 GMT):
`11:28:37 github.com/onsi/gomega/types: imported or required, but missing from Gopkg.lock's input-imports`
guoger (Thu, 17 Jan 2019 15:48:07 GMT):
@kostas ah, that was me being careless while pushing the previous [CR](https://gerrit.hyperledger.org/r/c/28778/), where `dep ensure` should be ran
guoger (Thu, 17 Jan 2019 15:48:12 GMT):
@yacovm you beat me on this
yacovm (Thu, 17 Jan 2019 15:48:16 GMT):
you're using forbidden magic (gomega)
guoger (Thu, 17 Jan 2019 15:48:27 GMT):
haha
kostas (Thu, 17 Jan 2019 15:55:16 GMT):
So verify-build failures are always legitimate user errors and not the CI dying on us.
guoger (Thu, 17 Jan 2019 16:55:03 GMT):
@kostas updated those CR, waiting for CI
guoger (Fri, 18 Jan 2019 09:41:46 GMT):
if a node is removed from consenter set *while it is disconnected*, how does it properly halt itself? if wouldn't be able to connect to other nodes at all due to the removal of certificate, right? even if it can connect, leader would not replicate data to it anyway... thoughts? @yacovm @kostas
guoger (Fri, 18 Jan 2019 09:42:23 GMT):
it is not a problem in etcd because this node is going to be shut down anyway...
guoger (Fri, 18 Jan 2019 09:42:23 GMT):
it is not a problem in etcd because this node is going to be shut down anyway... created https://jira.hyperledger.org/browse/FAB-13750 to track this. (it's currently under FAB-11863 (backlog)
guoger (Fri, 18 Jan 2019 09:42:23 GMT):
etcd doesn't deal with this problem because this node is going to be shut down anyway... created https://jira.hyperledger.org/browse/FAB-13750 to track this. (it's currently under FAB-11863 (backlog)
yacovm (Fri, 18 Jan 2019 10:18:54 GMT):
not sure what's the issue? I thought the `etcdraft.Chain` halts itself, @guoger
yacovm (Fri, 18 Jan 2019 10:19:15 GMT):
are you saying the node is removed and it never receives the config update about it?
yacovm (Fri, 18 Jan 2019 10:20:24 GMT):
it will keep trying to connect to other OSNs in that case
yacovm (Fri, 18 Jan 2019 10:21:30 GMT):
but we have the same problem in peers too - a peer doesn't detect its own eviction from an organization. It will keep trying to pull blocks despite not being eligible
yacovm (Fri, 18 Jan 2019 11:04:09 GMT):
I think we can solve this by having the node probe the system channel OSNs via deliver and seeing if it is a participant of the chain or not.
yacovm (Fri, 18 Jan 2019 11:04:13 GMT):
I'll take this on
yacovm (Fri, 18 Jan 2019 11:19:54 GMT):
will do it after I take care of the comm layer's Step RPC
guoger (Fri, 18 Jan 2019 12:19:37 GMT):
Cool. Just be mindful about the distinction between a newly added node and just removed node
guoger (Fri, 18 Jan 2019 12:21:03 GMT):
Hold on, how does system channel know if a node is part of an application channel?
guoger (Fri, 18 Jan 2019 12:22:13 GMT):
And what if a node is removed from sys but not app channel? Do we have explicit check to prevent that?
yacovm (Fri, 18 Jan 2019 14:39:24 GMT):
> Hold on, how does system channel know if a node is part of an application channel?
it doesn't. you need to ask every system channel *OSN* about the application channel
yacovm (Fri, 18 Jan 2019 14:40:25 GMT):
> And what if a node is removed from sys but not app channel? Do we have explicit check to prevent that?
We don't have a check for that, no.
yacovm (Fri, 18 Jan 2019 14:40:36 GMT):
if a node is removed from the system channel, it should be removed from all
guoger (Fri, 18 Jan 2019 14:54:44 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=Fx4h3rsv5LABn5hS5) @yacovm yeah... i have this in my mind: to remove a node from network, one needs to remove it from all the application channel, then remove it from system channel, and stop it.
guoger (Fri, 18 Jan 2019 14:54:44 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=Fx4h3rsv5LABn5hS5) @yacovm yeah... i have this in my mind: _to remove a node from network, one needs to remove it from all the application channel, then remove it from system channel, and stop it_ but not very willing to say it out :joy:
guoger (Fri, 18 Jan 2019 14:54:44 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=Fx4h3rsv5LABn5hS5) @yacovm yeah... i have this in my mind: _to remove a node from network, one needs to remove it from all the application channel, then remove it from system channel, and stop it._ but not very willing to say it out :joy:
guoger (Fri, 18 Jan 2019 14:54:44 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=Fx4h3rsv5LABn5hS5) @yacovm yeah... i have this in my mind: _to remove a node from network, one needs to remove it from all the application channel, then remove it from system channel_
guoger (Fri, 18 Jan 2019 14:56:48 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=TPNe5ACj7pB6huS3r) @yacovm i don't follow how this solves our problem here?
yacovm (Fri, 18 Jan 2019 15:00:56 GMT):
it solves the problem because the node can ask the system channel members whether the application channel he thinks he is part of, has its certificate in the latest config block
guoger (Fri, 18 Jan 2019 15:10:23 GMT):
but how does system channel access the config block (current consenter set) of a particular application channel?
guoger (Fri, 18 Jan 2019 15:11:03 GMT):
btw, have you seen this before [this](https://jenkins.hyperledger.org/job/fabric-verify-integration-tests-x86_64/4839/console)?
> 20:15:53 [e][peer-channel-create] Error: got unexpected status: BAD_REQUEST -- error authorizing update: error validating ReadSet: readset expected key [Group] /Channel/Application at version 0, but got version 1
guoger (Fri, 18 Jan 2019 15:11:03 GMT):
btw, have you seen this before [this](https://jenkins.hyperledger.org/job/fabric-verify-integration-tests-x86_64/4839/console)?
```
20:15:53 [e][peer-channel-create] Error: got unexpected status: BAD_REQUEST -- error authorizing update: error validating ReadSet: readset expected key [Group] /Channel/Application at version 0, but got version 1
```
yacovm (Fri, 18 Jan 2019 15:12:37 GMT):
it doesn't access.
yacovm (Fri, 18 Jan 2019 15:12:42 GMT):
the OSN will access it
yacovm (Fri, 18 Jan 2019 15:12:52 GMT):
it will poll the system channel OSNs
yacovm (Fri, 18 Jan 2019 15:13:06 GMT):
that means the channel is already created, Jay
guoger (Fri, 18 Jan 2019 15:14:38 GMT):
haha, i mean why does this fails onboard integration test?
guoger (Fri, 18 Jan 2019 15:14:38 GMT):
haha, i mean why does this fail onboard integration test?
guoger (Fri, 18 Jan 2019 15:20:15 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=skSeDCkjy3xas4k36) @yacovm bear with me. can you break this down? no hurry, it's late here anyway. Also, i feel it's a bit overkill. if certificate is refused, we do get error, right? and if we repeatedly encounter this error, it very likely means that we are banned from this channel, and should halt. operator could always remove and then add again this node, if that's not the case.
guoger (Fri, 18 Jan 2019 15:20:15 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=skSeDCkjy3xas4k36) @yacovm bear with me. can you break this down? no hurry, it's late here anyway. Also, i feel it's a bit overkill. if certificate is refused, we do get error, right? and if we repeatedly encounter this error, it very likely means that this node is banned from this channel, and should halt. operator could always remove and then add again this node, if that's not the case.
yacovm (Fri, 18 Jan 2019 15:23:36 GMT):
it's not that simple... because it can be that the nodes you are polling are the ones that are late
yacovm (Fri, 18 Jan 2019 15:23:40 GMT):
and don't know that you were added
yacovm (Fri, 18 Jan 2019 15:33:08 GMT):
so the idea i have in mind is - lets say you cannot reach any node in the channel and they all tell your TLS certificate is not in the channel.
yacovm (Fri, 18 Jan 2019 15:33:19 GMT):
you then just query all OSNs you know are in the system channel
yacovm (Fri, 18 Jan 2019 15:33:29 GMT):
each such an OSN returns to you its latest config block
yacovm (Fri, 18 Jan 2019 15:33:38 GMT):
if the latest config block they return to you is later than your own
yacovm (Fri, 18 Jan 2019 15:33:57 GMT):
and you get that config block, and see that you're not in the channel
yacovm (Fri, 18 Jan 2019 15:34:03 GMT):
then you halt the chain
kostas (Fri, 18 Jan 2019 17:08:42 GMT):
Good catch. I left a comment with some thoughts on FAB-13750.
kostas (Fri, 18 Jan 2019 17:09:17 GMT):
Let's try to review/merge as much of Jay's CR stack as we can in the meantime please.
guoger (Sat, 19 Jan 2019 00:20:40 GMT):
Ah, I guess I misinterpreted this to be _the latest config in systemchannel_ but you actually meant application channel, correct? @yacovm [ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=3hyzyFmGCjYSfxR6F)
yacovm (Sat, 19 Jan 2019 00:38:48 GMT):
yeah
guoger (Sat, 19 Jan 2019 02:05:30 GMT):
If a removed node notices that *all* other nodes in its channel *reject* its certificate, wouldn’t that be sufficient hint it’s been excluded from channel?
kostas (Sat, 19 Jan 2019 03:57:15 GMT):
@guoger: That’s the approach I suggested in the FAB you opened up.
kostas (Sat, 19 Jan 2019 04:03:46 GMT):
The TL;DR (see Yacov’s response in the thread) is that in order for us to be able to re-add that OSN to the channel on the future *and* be able to re-use a lot of the code that Yacov has written already, we need to append that config block that tells the OSN it’s not part of the ledger.
guoger (Sat, 19 Jan 2019 04:04:40 GMT):
Ah Just saw that
yacovm (Sat, 19 Jan 2019 08:17:36 GMT):
also, what if you have 8 nodes, and you remove 2 but they were offline and didn't get the config update. the 2 nodes come back online, and can talk to each other. that violates the *all* condition no?
yacovm (Sat, 19 Jan 2019 08:17:41 GMT):
@guoger
guoger (Sat, 19 Jan 2019 10:37:21 GMT):
True... what about rejected by quorum?
guoger (Sat, 19 Jan 2019 10:37:51 GMT):
Btw, can a node pull blocks from the channel it was removed from?
yacovm (Sat, 19 Jan 2019 10:50:45 GMT):
> Btw, can a node pull blocks from the channel it was removed from?
Uh, yeah... ? why not? removing the node only means you removed its certificate from the consenter set. you can still connect as a client.
kostas (Sat, 19 Jan 2019 10:55:32 GMT):
That is actually not the case I had in mind.
yacovm (Sat, 19 Jan 2019 10:55:34 GMT):
> True... what about rejected by quorum?
If you have, say - 3 nodes and 1 is brought offline. Now if you replace the certificates of the other 2 nodes (gradually, but due to periodic rotation), but also remove that offline node, and the offline node is brought back online later - it won't be able to connect to the quorum at all because the certificates have changed, and the TLS Dialer only trusts the specific certificates from the past
yacovm (Sat, 19 Jan 2019 10:56:39 GMT):
Basically - production grade permissioned blockchain is complex ;)
kostas (Sat, 19 Jan 2019 10:56:50 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=SqZ4zKjSR4odpFAL2) @ Consider the chase where you remove a node because that org leaves the consortium.
kostas (Sat, 19 Jan 2019 10:56:50 GMT):
Consider the case where you remove a node because that org leaves the consortium.
yacovm (Sat, 19 Jan 2019 10:57:28 GMT):
then you get a forbidden status
yacovm (Sat, 19 Jan 2019 10:57:33 GMT):
and you know you're outside
yacovm (Sat, 19 Jan 2019 10:57:46 GMT):
onboarding actually has this as a use case, which is tested in UTs
yacovm (Sat, 19 Jan 2019 10:58:01 GMT):
and in integration test
yacovm (Sat, 19 Jan 2019 10:58:59 GMT):
(I make a policy that says that no one can pull blocks, but only the onboarded OSN is effected by it :joy: because the other OSNs don't call deliver by use Raft )
guoger (Sun, 20 Jan 2019 03:43:45 GMT):
> but also remove that offline node, and the offline node is brought back online later - it won't be able to connect to the quorum at all because the certificates have changed, and the TLS Dialer only trusts the specific certificates from the past
a seemingly irrelevant question: if the offline node is *not* removed from the channel, how will it reconnect to other nodes when brought back online?
yacovm (Sun, 20 Jan 2019 08:42:03 GMT):
if their certificates/endpoints changed, it won't. that's why we need to make the change i proposed to make the OSN pull the blocks to see what's going on, and also have the administrators remove it and re-add it back.
guoger (Sun, 20 Jan 2019 14:49:27 GMT):
wait.... a second look at my question gives me pause.. it should still be able to connect to network, because its own certificate is not removed from consenter set, right?
back to the scenario you described, if offline node is removed and not being able to connect to other nodes, i still don't get the necessity of it learning this from latest config block.. :(
yacovm (Sun, 20 Jan 2019 16:32:38 GMT):
because you need to make it understand that it is out of the chain. Once it is disconnected and removed it can't ask for blocks via Raft so nothing would tell it about its eviction.
guoger (Tue, 22 Jan 2019 08:10:32 GMT):
hmm.... why does CI give me `No Builds Executed`?
ycarmel (Tue, 22 Jan 2019 08:46:55 GMT):
Has joined the channel.
incarose (Wed, 23 Jan 2019 00:23:55 GMT):
Has joined the channel.
allan.gulley (Wed, 23 Jan 2019 03:29:36 GMT):
Has joined the channel.
guoger (Wed, 23 Jan 2019 10:39:08 GMT):
updated https://gerrit.hyperledger.org/r/c/28918/ to point to sub-task. Another problem is, we couldn't increase tick interval beyond 250ms. That would result in a 5s leaderless period, and prevent `Deliver` api to return genesis block of newly created channel. and our {{peer channel create}} has default timeout of 5s.
One way to solve this, is to explicitly call `Campaign` on a raft node upon boot, to proactively start election, instead of waiting for ElectionTimeout. However, we need to decide how to have nodes decide on which node to start election... and i say we take the easy way for now: 1) enlarge peer timeout, or 2) leave it to 200/250ms
guoger (Wed, 23 Jan 2019 10:39:08 GMT):
updated https://gerrit.hyperledger.org/r/c/28918/ to point to sub-task. Another problem is, we couldn't increase tick interval beyond 250ms. That would result in a 5s leaderless period, and prevent `Deliver` api to return genesis block of newly created channel. and our {{peer channel create}} has default timeout of 5s.
One way to solve this, is to explicitly call `Campaign` on a raft node upon boot, to proactively start election, instead of waiting for ElectionTimeout. However, we need to decide how to have nodes decide on which node to start election... and i say we take the easy way for now: 1) increase peer timeout, or 2) leave it to 200/250ms
guoger (Wed, 23 Jan 2019 10:39:13 GMT):
wdyt @yacovm
guoger (Wed, 23 Jan 2019 10:43:06 GMT):
or we could have our own random timer at startup, and call `Campaign`
yacovm (Wed, 23 Jan 2019 11:14:34 GMT):
let's jut fix the peer CLI to have a bigger default timeout
yacovm (Wed, 23 Jan 2019 11:15:27 GMT):
The election period is now longer, any way you look at it
yacovm (Wed, 23 Jan 2019 11:15:46 GMT):
even if you start campaigning immediately it can only halve the timeout
tock (Wed, 23 Jan 2019 12:06:57 GMT):
I know this is unrelated to the orderer, but: CI unit test failed
```
12:59:31 ok github.com/hyperledger/fabric/core/admin 0.406s coverage: 98.8% of statements
12:59:31 ==================
12:59:31 WARNING: DATA RACE
12:59:31 Write at 0x000001d3e110 by goroutine 26:
12:59:31 github.com/hyperledger/fabric/core/cclifecycle_test.newLogRecorder()
12:59:31 /w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/core/cclifecycle/lifecycle_test.go:464 +0x99
12:59:31 github.com/hyperledger/fabric/core/cclifecycle_test.TestMetadata()
12:59:31 /w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/core/cclifecycle/lifecycle_test.go:344 +0x53
12:59:31 testing.tRunner()
12:59:31 /opt/go/go1.11.1.linux.amd64/src/testing/testing.go:827 +0x162
12:59:31
12:59:31 2019-01-23 10:58:34.887 UTC [discovery.lifecycle] DeployedChaincodes -> ERRO 033 Chaincode cc1 is listed in LSCC as cc2
12:59:31 2019-01-23 10:58:34.888 UTC [discovery.lifecycle] DeployedChaincodes -> INFO 034 Chaincode cc1 isn't instantiated
12:59:31 FAIL
12:59:31 coverage: 99.3% of statements
```
any ideas who's in charge of that?
tock (Wed, 23 Jan 2019 12:06:57 GMT):
I know this is unrelated to the orderer, but: CI unit test failed
```
12:59:31 ok github.com/hyperledger/fabric/core/admin 0.406s coverage: 98.8% of statements
12:59:31 ==================
12:59:31 WARNING: DATA RACE
12:59:31 Write at 0x000001d3e110 by goroutine 26:
12:59:31 github.com/hyperledger/fabric/core/cclifecycle_test.newLogRecorder()
12:59:31 /w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/core/cclifecycle/lifecycle_test.go:464 +0x99
12:59:31 github.com/hyperledger/fabric/core/cclifecycle_test.TestMetadata()
12:59:31 /w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/core/cclifecycle/lifecycle_test.go:344 +0x53
12:59:31 testing.tRunner()
12:59:31 /opt/go/go1.11.1.linux.amd64/src/testing/testing.go:827 +0x162
...
12:59:31 2019-01-23 10:58:34.887 UTC [discovery.lifecycle] DeployedChaincodes -> ERRO 033 Chaincode cc1 is listed in LSCC as cc2
12:59:31 2019-01-23 10:58:34.888 UTC [discovery.lifecycle] DeployedChaincodes -> INFO 034 Chaincode cc1 isn't instantiated
12:59:31 FAIL
12:59:31 coverage: 99.3% of statements
12:59:31 FAIL github.com/hyperledger/fabric/core/cclifecycle 0.523s
```
any ideas who's in charge of that?
tock (Wed, 23 Jan 2019 12:06:57 GMT):
I know this is unrelated to the orderer, but: CI unit test failed
```
12:59:31 ok github.com/hyperledger/fabric/core/admin 0.406s coverage: 98.8% of statements
12:59:31 ==================
12:59:31 WARNING: DATA RACE
12:59:31 Write at 0x000001d3e110 by goroutine 26:
12:59:31 github.com/hyperledger/fabric/core/cclifecycle_test.newLogRecorder()
12:59:31 /w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/core/cclifecycle/lifecycle_test.go:464 +0x99
12:59:31 github.com/hyperledger/fabric/core/cclifecycle_test.TestMetadata()
12:59:31 /w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/core/cclifecycle/lifecycle_test.go:344 +0x53
12:59:31 testing.tRunner()
12:59:31 /opt/go/go1.11.1.linux.amd64/src/testing/testing.go:827 +0x162
...
12:59:31 2019-01-23 10:58:34.887 UTC [discovery.lifecycle] DeployedChaincodes -> ERRO 033 Chaincode cc1 is listed in LSCC as cc2
12:59:31 2019-01-23 10:58:34.888 UTC [discovery.lifecycle] DeployedChaincodes -> INFO 034 Chaincode cc1 isn't instantiated
12:59:31 FAIL
12:59:31 coverage: 99.3% of statements
12:59:31 FAIL github.com/hyperledger/fabric/core/cclifecycle 0.523s
```
any ideas who's in charge of that?
tock (Wed, 23 Jan 2019 12:06:57 GMT):
I know this is unrelated to the orderer, but: CI unit test failed
```
12:59:31 ok github.com/hyperledger/fabric/core/admin 0.406s coverage: 98.8% of statements
12:59:31 ==================
12:59:31 WARNING: DATA RACE
12:59:31 Write at 0x000001d3e110 by goroutine 26:
12:59:31 github.com/hyperledger/fabric/core/cclifecycle_test.newLogRecorder()
12:59:31 /w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/core/cclifecycle/lifecycle_test.go:464 +0x99
12:59:31 github.com/hyperledger/fabric/core/cclifecycle_test.TestMetadata()
12:59:31 /w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/core/cclifecycle/lifecycle_test.go:344 +0x53
12:59:31 testing.tRunner()
12:59:31 /opt/go/go1.11.1.linux.amd64/src/testing/testing.go:827 +0x162
...
12:59:31 2019-01-23 10:58:34.887 UTC [discovery.lifecycle] DeployedChaincodes -> ERRO 033 Chaincode cc1 is listed in LSCC as cc2
12:59:31 2019-01-23 10:58:34.888 UTC [discovery.lifecycle] DeployedChaincodes -> INFO 034 Chaincode cc1 isn't instantiated
12:59:31 FAIL
12:59:31 coverage: 99.3% of statements
12:59:31 FAIL github.com/hyperledger/fabric/core/cclifecycle 0.523s
```
any ideas who's in charge of that?
guoger (Wed, 23 Jan 2019 12:15:04 GMT):
@tock i create FAB-13843 for this, made a minor attempt to fix but gave up :(
guoger (Wed, 23 Jan 2019 12:15:35 GMT):
i guess @sykesm or @jyellick could take a look?
kostas (Wed, 23 Jan 2019 15:05:52 GMT):
> However, we need to decide how to have nodes decide on which node to start election...
kostas (Wed, 23 Jan 2019 15:05:58 GMT):
@guoger: Can you expand on this?
jyellick (Wed, 23 Jan 2019 17:13:46 GMT):
What is it that I would take a look at? The core/cclifecycle data race? Assuming it's new... I would go to whoever introduced it. I know @yacovm recently made some changes in there (though not sure if they're the culprit)
yacovm (Wed, 23 Jan 2019 17:22:20 GMT):
I usually am, @jyellick
yacovm (Wed, 23 Jan 2019 17:22:25 GMT):
(the culprit)
yacovm (Wed, 23 Jan 2019 17:23:04 GMT):
I put the JIRA on my tab
jyellick (Wed, 23 Jan 2019 17:23:27 GMT):
Thanks @yacovm
guoger (Thu, 24 Jan 2019 02:29:55 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=zght732vhXdyRvhGj) @kostas what i have in mind is, when a chain has just started (`Start` is called), we spawn a background go routine doing something like this:
```
select {
case <-time.After(randomTimeout):
c.logger.Infof("Proactively start campaign")
c.node.Campaign()
case <-c.electedC:
fallthrough
case <-c.doneC:
return
}
```
tock (Thu, 24 Jan 2019 09:06:41 GMT):
I know this is unrelated to the orderer, but: CI unit test failed
```
12:59:31 ok github.com/hyperledger/fabric/core/admin 0.406s coverage: 98.8% of statements
12:59:31 ==================
12:59:31 WARNING: DATA RACE
12:59:31 Write at 0x000001d3e110 by g
...
12:59:31 2019-01-23 10:58:34.887 UTC [discovery.lifecycle] DeployedChaincodes -> ERRO 033 Chaincode cc1 is listed in LSCC as cc2
12:59:31 2019-01-23 10:58:34.888 UTC [discovery.lifecycle] DeployedChaincodes -> INFO 034 Chaincode cc1 isn't instantiated
12:59:31 FAIL
12:59:31 coverage: 99.3% of statements
12:59:31 FAIL github.com/hyperledger/fabric/core/cclifecycle 0.523s
```
any ideas who's in charge of that?
tock (Sun, 27 Jan 2019 08:24:29 GMT):
Again, unrelated to orderer but bothering all of us regardless; this failure in unit-test:
```09:50:50 Running Suite: Grpclogging Suite
09:50:50 ================================
09:50:50 Random Seed: 1548575439
09:50:50 Will run 31 of 31 specs
09:50:50
09:50:50 •••••••••••••••••••••••
09:50:50 ------------------------------
09:50:50 • Failure [0.363 seconds]
09:50:50 Server
09:50:50 /w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/common/grpclogging/server_test.go:36
09:50:50 StreamServerInterceptor
09:50:50 /w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/common/grpclogging/server_test.go:356
09:50:50 when options are used
09:50:50 /w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/common/grpclogging/server_test.go:579
09:50:50 uses the levels returned by the levelers [It]
09:50:50 /w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/common/grpclogging/server_test.go:636
09:50:50
09:50:50 Expected
09:50:50 <[]observer.LoggedEntry | len:0, cap:0>: []
09:50:50 to have length 1
09:50:50
09:50:50 /w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/common/grpclogging/server_test.go:638
09:50:50 ------------------------------
09:50:50 •••••••
09:50:50
09:50:50 Summarizing 1 Failure:
09:50:50
09:50:50 [Fail] Server StreamServerInterceptor when options are used [It] uses the levels returned by the levelers
09:50:50 /w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/common/grpclogging/server_test.go:638
09:50:50
09:50:50 Ran 31 of 31 Specs in 11.297 seconds
09:50:50 FAIL! -- 30 Passed | 1 Failed | 0 Pending | 0 Skipped
09:50:50 --- FAIL: TestGrpclogging (11.30s)
09:50:50 FAIL
09:50:50 coverage: 100.0% of statements
09:50:50 FAIL github.com/hyperledger/fabric/common/grpclogging 11.372s
```
tock (Sun, 27 Jan 2019 08:25:08 GMT):
any ideas who's in charge of that?
tock (Sun, 27 Jan 2019 09:48:48 GMT):
https://jira.hyperledger.org/browse/FAB-12777
C0rWin (Sun, 27 Jan 2019 12:38:24 GMT):
@sykesm ^^^
kostas (Sun, 27 Jan 2019 21:58:25 GMT):
@tock: Search JIRA for all items labeled as `flakes`. This shows https://jira.hyperledger.org/browse/FAB-13867 (which is already assigned to Matt). Can you post a comment there with the link to the CI failure?
guoger (Mon, 28 Jan 2019 01:59:39 GMT):
UT is passed for https://gerrit.hyperledger.org/r/c/28918/, although green label was not added :thinking:
yacovm (Mon, 28 Jan 2019 06:42:29 GMT):
@guoer i rebased my change set on top of your chain
yacovm (Mon, 28 Jan 2019 06:43:06 GMT):
Please pile on top of mine if add more
yacovm (Mon, 28 Jan 2019 06:43:31 GMT):
Ok? :)
guoger (Mon, 28 Jan 2019 06:58:01 GMT):
@yacovm sure :) actually i wanted to ask, should i rebase yours if i make change to the current stack? (and i'm about to, for the conclusion we reached on snapshot pruning)
tock (Mon, 28 Jan 2019 07:15:51 GMT):
@guoger just had a failure in integration:
```
08:42:36 Summarizing 2 Failures:
08:42:36
08:42:36 [Fail] EndToEnd Crash Fault Tolerance when orderer stops and restarts [It] keeps network up and running
08:42:36 /w/workspace/fabric-verify-integration-tests-x86_64/gopath/src/github.com/hyperledger/fabric/integration/e2e/cft_test.go:121
08:42:36
08:42:36 [Fail] EndToEnd Crash Fault Tolerance when an orderer is behind the latest snapshot on leader [It] catches up using the block stored in snapshot
08:42:36 /w/workspace/fabric-verify-integration-tests-x86_64/gopath/src/github.com/hyperledger/fabric/integration/e2e/cft_test.go:172
08:42:36
08:42:36 Ran 15 of 15 Specs in 623.478 seconds
08:42:36 FAIL! -- 13 Passed | 2 Failed | 0 Pending | 0 Skipped
08:42:36 --- FAIL: TestEndToEnd (623.48s)
```
https://jenkins.hyperledger.org/job/fabric-verify-integration-tests-x86_64/5065/console
guoger (Mon, 28 Jan 2019 08:11:28 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=4xzqcSXspdF6cb7hg) @tock Not really sure what's going on there (for both of them) ... the problem happens here:
```
peerProc = ifrit.Invoke(peerGroup)
Eventually(peerProc.Ready()).Should(BeClosed())
```
test waits for peer to start. However, I don't see peer logs in console at all... filed https://jira.hyperledger.org/browse/FAB-13914 to track this
guoger (Mon, 28 Jan 2019 08:11:28 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=4xzqcSXspdF6cb7hg) @tock Not really sure what's going on there (for both of them) ... the problem happens here:
```
peerProc = ifrit.Invoke(peerGroup)
Eventually(peerProc.Ready()).Should(BeClosed())
```
test waits for peer to start. However, I don't see peer logs in console at all... filed https://jira.hyperledger.org/browse/FAB-13914 to track this
tock (Mon, 28 Jan 2019 08:15:01 GMT):
I rerun the test... it overwrites the log :rolling_eyes:
tock (Mon, 28 Jan 2019 08:15:31 GMT):
it will happen again, I am sure ;-)
yacovm (Mon, 28 Jan 2019 08:17:19 GMT):
@guoger
> actually i wanted to ask, should i rebase yours if i make change to the current stack? (and i'm about to, for the conclusion we reached on snapshot pruning)
are you making changes below my CR or above it? if below then don't bother... I'll just rebase eventually again when I'm done. If above then you can just rebase on top
guoger (Mon, 28 Jan 2019 08:17:54 GMT):
gotcha. I meant below
guoger (Mon, 28 Jan 2019 09:56:48 GMT):
a general question: if i'm filing a JIRA for flaky test found in CI, what `affects version` should i tag it?
yacovm (Mon, 28 Jan 2019 10:28:56 GMT):
2.0
edisinovcic (Mon, 28 Jan 2019 13:15:53 GMT):
Has joined the channel.
guoger (Tue, 29 Jan 2019 02:49:08 GMT):
may one of you @kostas @tock put a summary of the conclusion you reached yesterday to https://jira.hyperledger.org/browse/FAB-13247? thx, i think i didn't fully get your solution...
kostas (Wed, 30 Jan 2019 03:15:14 GMT):
I'll let @tock chime in on this.
tock (Wed, 30 Jan 2019 07:50:08 GMT):
W.r.t https://jira.hyperledger.org/browse/FAB-13247 (recovery from crashes) - I will take a minimalistic approach, but will take care of recovery from a failure (we have to). I think my dilemma was not whether to do it or not, but really what kind of assumptions I could take on the behavior of the user (admin). If the user is rational and not byzantine, i.e. he follows protocol when submitting the sequence of config transactions, then I can design a recovery mechanism that is relatively simple. This will address one main concern - an OSN that crashes before or during migration, and recovers during or after migration. What is not covered is (some) user behavior that is documented to be bad - submitting conflicting migration commands in quick succession from multiple OSN's, and things like that. The code protects against most of these scenarios, but not against all.
I documented the protocol for migration here: https://docs.google.com/document/d/1g_M8KO9cdbYMLBsscHd6ffv-X_ZRLeHvDcIEoV7YXUE/edit?usp=sharing
And I will update this document as I make progress, to include the abort path, and a recovery from failures.
tock (Wed, 30 Jan 2019 07:50:08 GMT):
W.r.t https://jira.hyperledger.org/browse/FAB-13247 (recovery from crashes) - I will take a minimalistic approach, but will take care of recovery from a failure (we have to). I think my dilemma was not whether to do it or not, but really what kind of assumptions I could take on the behavior of the user (admin). If the user is rational and not byzantine, i.e. he follows protocol when submitting the sequence of config transactions, then I can design a recovery mechanism that is relatively simple. This will address one main concern - an OSN that crashes before or during migration, and recovers during or after migration. What is not covered is (some) user behavior that is documented to be bad - submitting conflicting migration commands in quick succession from multiple OSN's, and things like that. The code protects against most of these scenarios, but not against all.
I documented the protocol for migration here: https://docs.google.com/document/d/1g_M8KO9cdbYMLBsscHd6ffv-X_ZRLeHvDcIEoV7YXUE/edit?usp=sharing
And I will update this document as I make progress, to include the abort path, and a recovery from failures. There is a link to this document from the main story https://jira.hyperledger.org/browse/FAB-12777 as well.
guoger (Wed, 30 Jan 2019 09:54:12 GMT):
> the behavior of the user (admin)
My take is, they are rational, dumb, and trying to play smart. So as long as we can recover to previous state (kafka), i think we are fine.
> then I can design a recovery mechanism that is relatively simple
guoger (Wed, 30 Jan 2019 09:54:12 GMT):
> the behavior of the user (admin)
My take is, they are rational, dumb, and trying to play smart. So as long as we can recover to previous state (kafka), i think we are fine.
> then I can design a recovery mechanism that is relatively simple
If at some point of time, you can have a small writeup about this mechanism, i'd appreciate it! cc @tock
tock (Wed, 30 Jan 2019 10:50:37 GMT):
I am working on it in the Jira.
tock (Wed, 30 Jan 2019 10:50:37 GMT):
I am working on it in the Jira (the writeup). I'll let you know when i am done
tock (Wed, 30 Jan 2019 10:52:49 GMT):
I agree that users sometime make mistakes. we'll have to clearly document what we expect and also probably make a tool (maybe a bunch of scripts) to help them do it right
tock (Wed, 30 Jan 2019 10:58:46 GMT):
In addition, the START command stops all traffic to the orderers, apart from migration related config updates. Waiting for the start to be committed, an then waiting for the Kafka pipeline to drain, gives a good backup point.
guoger (Wed, 30 Jan 2019 15:40:50 GMT):
@yacovm i think your comm CR is not part of my stack now (and i don't think it needs to be). But if you ever want/need, just insert it to the bottom/top/middle of my stack as you see fit.
guoger (Thu, 31 Jan 2019 14:17:25 GMT):
I slightly doubt that follower forwarding tx to leader would become the bottleneck, but that's definitely a great point!
guoger (Thu, 31 Jan 2019 14:17:25 GMT):
I slightly doubt that follower forwarding tx to leader would become the bottleneck with raft, but that's definitely a great point!
guoger (Thu, 31 Jan 2019 14:17:44 GMT):
as my current benchmark suggests, disk io seems to be a constraint
guoger (Thu, 31 Jan 2019 14:18:03 GMT):
(which is kind of expected because raft protocol is io intensive by definition)
kostas (Thu, 31 Jan 2019 14:35:47 GMT):
We've got 4 CRs with one +2 already that we should try to merge today:
https://gerrit.hyperledger.org/r/c/28850/
https://gerrit.hyperledger.org/r/c/28998/
https://gerrit.hyperledger.org/r/c/29034/
https://gerrit.hyperledger.org/r/c/28892/
guoger (Fri, 01 Feb 2019 04:11:21 GMT):
to avoid 5-10s leaderless in a newly created channel (due to prolonged tick interval), i suggest that we can have the node with *smallest* raft id to proactively start campaign via raft api, *iff* it's a new cluster (not rejoin, not reboot). If, for some reasons, the node with smallest id is absent, we just regress to normal case, where other nodes wait for election timeout. but i think we should've got most of cases covered this way. Of course, this can be backlogged with lower priority. I'm just trying to see if it's worth a jira in our backlog, and i'm not super comfortable with deterministic 5-10s leaderless when a new channel is created.
guoger (Fri, 01 Feb 2019 04:11:21 GMT):
to avoid 5-10s leaderless in a newly created channel (due to prolonged tick interval), i suggest that we can have the node with *smallest* raft id to proactively start campaign via raft api, *iff* it's a new cluster (not rejoin, not reboot). If, for some reasons, the node with smallest id is absent, we just regress to normal case, where other nodes wait for election timeout. but i think we should've got most of cases covered this way. Of course, this can be backlogged with lower priority. I'm just trying to see if it's worth a jira in our backlog, and i'm not super comfortable with deterministic 5-10s leaderless when a new channel is created.
guoger (Fri, 01 Feb 2019 04:11:30 GMT):
wdyt @yacovm @kostas
adarshsaraf123 (Fri, 01 Feb 2019 05:05:23 GMT):
@guoger to pitch in here, is it not possible to trigger a leader election artificially in the case that this is a new channel like what you have done in the tests?
guoger (Fri, 01 Feb 2019 05:18:20 GMT):
but who's going to trigger it? and i don't think we should add an api to `Chain`
yacovm (Fri, 01 Feb 2019 07:44:19 GMT):
@guoger sounds fine to me... but can we add a few lines in some integration test to actually see that it works?
tock (Sun, 03 Feb 2019 10:39:49 GMT):
There is a unit test failure on etcdraft:
12:35:42 FAIL github.com/hyperledger/fabric/orderer/consensus/etcdraft 1200.460s
https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/8591/console
kostas (Sun, 03 Feb 2019 14:19:59 GMT):
@tock: Is there a JIRA for it?
kostas (Sun, 03 Feb 2019 16:34:15 GMT):
> to avoid 5-10s leaderless in a newly created channel (due to prolonged tick interval), i suggest that we can have the node with *smallest* raft id to proactively start campaign via raft api, *iff* it's a new cluster (not rejoin, not reboot). If, for some reasons, the node with smallest id is absent, we just regress to normal case, where other nodes wait for election timeout. but i think we should've got most of cases covered this way. Of course, this can be backlogged with lower priority. I'm just trying to see if it's worth a jira in our backlog, and i'm not super comfortable with deterministic 5-10s leaderless when a new channel is created.
kostas (Sun, 03 Feb 2019 16:34:47 GMT):
@guoger: Sorry for missing this earlier. Doesn't this give an advantage to the node with the smallest ID?
kostas (Sun, 03 Feb 2019 16:36:09 GMT):
*If* it does, I would advise against it. In general we want the spread of leaders to be as diversified as possible. I get that the leader is not static, but I can see how users may not be big fans of this choice.
kostas (Sun, 03 Feb 2019 16:36:30 GMT):
The creation of a channel is a one-off process; I think a 10-second delay is OK at the end of the day.
tock (Sun, 03 Feb 2019 18:22:20 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=88aae33b-86bb-405d-a0a9-3040055532bc) @kostas no
tock (Sun, 03 Feb 2019 18:26:10 GMT):
13:29:50 FAIL github.com/hyperledger/fabric/orderer/consensus/etcdraft 1200.537s
https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/8592/console
Oops, it happened again ;-)
kostas (Sun, 03 Feb 2019 20:25:00 GMT):
@tock: Roger, can you please open up a JIRA for it? Add it as a child of this story: https://jira.hyperledger.org/browse/FAB-13371
guoger (Mon, 04 Feb 2019 02:58:55 GMT):
@tock is it rebased on master?
adarshsaraf123 (Wed, 06 Feb 2019 10:05:20 GMT):
@guoger @kostas @yacovm An early WIP on the metrics thread without the tests to see if there are any comments:
https://gerrit.hyperledger.org/r/#/c/29153/
adarshsaraf123 (Wed, 06 Feb 2019 10:05:34 GMT):
Would appreciate a quick look at it.
adarshsaraf123 (Wed, 06 Feb 2019 10:07:01 GMT):
I also need to take care of rebasing it on the right stack. @guoger Can you point me to the right tip for this?
adarshsaraf123 (Wed, 06 Feb 2019 10:07:01 GMT):
I also need to take care of rebasing it on the right stack. @guoger Can you also point me to the right tip for this? Thanks :)
guoger (Wed, 06 Feb 2019 16:37:30 GMT):
@adarshsaraf123 skimmed through your CR and i think i looks ok (I'll review once it's ready). i say you don't worry about rebasing for now. i'll resolve the conflicts once mine is done.
guoger (Wed, 06 Feb 2019 16:37:43 GMT):
another question, are we going to add metrics for kafka-based chain?
jyellick (Wed, 06 Feb 2019 17:37:37 GMT):
Would someone mind taking a look at this and telling me if it looks like a known flake? https://jenkins.hyperledger.org/job/fabric-verify-integration-tests-x86_64/5419/console
adarshsaraf123 (Wed, 06 Feb 2019 17:45:35 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=Qg8veQYNqfT6wEva3) @guoger @jyellick had implemented some metrics for kafka-based chains that mostly wraps metrics implemented by Sarama. Link to the same: https://gerrit.hyperledger.org/r/#/c/27741/
yacovm (Wed, 06 Feb 2019 17:45:36 GMT):
lol, all tests but the reconfiguration suite failed
yacovm (Wed, 06 Feb 2019 17:46:05 GMT):
no @jyellick I've never seen 6 out of 11 tests fail at the same time
jyellick (Wed, 06 Feb 2019 17:46:43 GMT):
Okay -- I figured it was something I broke, thanks for confirming
guoger (Wed, 06 Feb 2019 17:47:40 GMT):
?
yacovm (Wed, 06 Feb 2019 17:47:57 GMT):
how can you break something? you don't touch Raft @jyellick
yacovm (Wed, 06 Feb 2019 17:48:03 GMT):
the error there was that there is no raft leader
jyellick (Wed, 06 Feb 2019 17:48:25 GMT):
Actually, I've been working on Raft for the past few days, implementing those changes to `configtxgen` :slight_smile:
guoger (Thu, 07 Feb 2019 14:31:53 GMT):
@tock could you explain the purpose of having `Context` storing the block number? it's not documented in both design docs. thx in advance!
tock (Thu, 07 Feb 2019 14:47:28 GMT):
It is in order to collate a context, commit and an abort to the start that opened the migration. It is also a way to force the user to wait for start to commit in order to submit the context (you have to know the height).
tock (Thu, 07 Feb 2019 14:48:54 GMT):
It is enforced in the code, so that the context / commit / abort must match the start context. the context can be thought of as a "transaction id", for the big nested transaction that is migration.
kostas (Thu, 07 Feb 2019 14:49:47 GMT):
Ah, I remember you mentioning this in the presentation. It has no use as a rollback piece of info, does it?
kostas (Thu, 07 Feb 2019 14:50:21 GMT):
In the sense that no piece of code is reading it and says: I should roll back to the config, as noted in migration context (block height) X.
tock (Thu, 07 Feb 2019 14:51:21 GMT):
When I say Rollback, I mean the procedure that would be taken when you restart into raft and find out to your horror that the cluster does not form, or crash...
tock (Thu, 07 Feb 2019 14:55:54 GMT):
This can happen if the user puts the wrong certificates in the raft-metadata, for example, and we don't spot it before commit. At that point, What I am planning on doing, it to instruct the user to manually swap the bootstrap files - put the original kafka genesis block back in place, and restart. I add a small piece of code that will detect this kind of restart (easy - last config block has type=raft, whereas bootstrap has type=kafka), and enter into a "special" state that would allow the user to do an abort sequence.
tock (Thu, 07 Feb 2019 14:55:54 GMT):
This can happen if the user puts the wrong certificates in the raft-metadata, for example, and we don't spot it before commit. At that point, What I am planning on doing, it to instruct the user to manually swap the bootstrap files - put the original kafka genesis block back in place, and restart. I will add a small piece of code that will detect this kind of restart (easy - last config block has type=raft, whereas bootstrap has type=kafka), and enter into a "special" state that would allow the user to do an abort sequence.
guoger (Thu, 07 Feb 2019 15:12:41 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=HgcvTh9dbQqmSBjmJ) @tock ahh, got it. I was struggling to understand the txid vs context during the presentation...
guoger (Thu, 07 Feb 2019 15:13:29 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=EHh77jTdp7n9tv7et) @tock i know this case is very unlikely, but when you say "collate", does it mean that we are prepared for the cases where `context` number among them _don't check_?
guoger (Fri, 08 Feb 2019 16:02:41 GMT):
A very preliminary benchmark:
```
Type: kafka Nodes: 3 Broadcast Clients: 15 Total Tx: 3000 Elapsed Time: 10.66s Write tps: 281.3 tx/s
Type: etcdraft Nodes: 3 Broadcast Clients: 15 Total Tx: 3000 Elapsed Time: 4.95s Write tps: 605.9 tx/s
```
guoger (Fri, 08 Feb 2019 16:02:41 GMT):
A very preliminary benchmark (batchsize == 1):
```
Type: kafka Nodes: 3 Broadcast Clients: 15 Total Tx: 3000 Elapsed Time: 10.66s Write tps: 281.3 tx/s
Type: etcdraft Nodes: 3 Broadcast Clients: 15 Total Tx: 3000 Elapsed Time: 4.95s Write tps: 605.9 tx/s
```
guoger (Fri, 08 Feb 2019 16:03:10 GMT):
(all running locally on my OSX)
guoger (Fri, 08 Feb 2019 16:03:10 GMT):
(all running locally on my OSX, where wal sync is super slow)
kostas (Fri, 08 Feb 2019 16:39:51 GMT):
Suspiciously good?
kostas (Fri, 08 Feb 2019 16:39:51 GMT):
@guoger: Numbers looking suspiciously good? :grin:
kostas (Fri, 08 Feb 2019 16:44:28 GMT):
Two CRs that I _think_ look good for merging:
kostas (Fri, 08 Feb 2019 16:44:30 GMT):
https://gerrit.hyperledger.org/r/c/29180/
kostas (Fri, 08 Feb 2019 16:44:34 GMT):
https://gerrit.hyperledger.org/r/c/29152/
guoger (Fri, 08 Feb 2019 16:54:21 GMT):
That’s why I say _very preliminary_ ;-)
guoger (Fri, 08 Feb 2019 16:54:54 GMT):
I’ll take a closer look later
yacovm (Fri, 08 Feb 2019 17:19:58 GMT):
I say that @guoger is a bit biased
yacovm (Fri, 08 Feb 2019 17:20:08 GMT):
he must've faked the results ;)
yacovm (Fri, 08 Feb 2019 17:20:19 GMT):
(just kidding)
yacovm (Fri, 08 Feb 2019 17:21:36 GMT):
@guoger what is the setup of the benchmark?
guoger (Fri, 08 Feb 2019 23:13:51 GMT):
@yacovm here: https://gerrit.hyperledger.org/r/c/27846/9
guoger (Fri, 08 Feb 2019 23:49:59 GMT):
and maybe that could help with debugging memory leak...
yacovm (Sat, 09 Feb 2019 00:03:40 GMT):
a test that runs several seconds?
guoger (Sat, 09 Feb 2019 00:03:54 GMT):
and i think 10 is probably too harsh for default `SendBufferSize`
yacovm (Sat, 09 Feb 2019 00:04:10 GMT):
do you see dropped messages?
guoger (Sat, 09 Feb 2019 00:04:15 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=bjEWhqRCB73xYac9E) @yacovm you can modify parameters to have it run for minutes/hours
yacovm (Sat, 09 Feb 2019 00:04:29 GMT):
but you put in the CR - no compaction
guoger (Sat, 09 Feb 2019 00:04:30 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=vzubZjvEvf9shrkkG) @yacovm a lot...
yacovm (Sat, 09 Feb 2019 00:04:37 GMT):
the memory will explode
guoger (Sat, 09 Feb 2019 00:04:49 GMT):
you can change that too
yacovm (Sat, 09 Feb 2019 00:04:52 GMT):
yep
yacovm (Sat, 09 Feb 2019 00:05:24 GMT):
> a lot...
this is interesting. how many blocks are created?
guoger (Sat, 09 Feb 2019 00:05:57 GMT):
thousands
yacovm (Sat, 09 Feb 2019 00:06:48 GMT):
so we have 3,000 transactions
yacovm (Sat, 09 Feb 2019 00:06:54 GMT):
and thousands of blocks are created?
yacovm (Sat, 09 Feb 2019 00:06:58 GMT):
doesn't make sense
yacovm (Sat, 09 Feb 2019 00:09:03 GMT):
```
for _, p := range cfg.Profiles {
p.BatchSize.MaxMessageCount = MaxMessageCount
p.SnapshotInterval = SnapshotInterval
```
yacovm (Sat, 09 Feb 2019 00:09:10 GMT):
why are we putting 1 message in a batch?
yacovm (Sat, 09 Feb 2019 00:09:10 GMT):
why are we putting 1 message in a batch? `MaxMessageCount = 1 // MaxMessageCount in BatchSize`
yacovm (Sat, 09 Feb 2019 00:09:18 GMT):
this isn't a realistic scenario....
yacovm (Sat, 09 Feb 2019 00:09:38 GMT):
Raft needs big blocks, not blocks with a single envelope.
yacovm (Sat, 09 Feb 2019 00:10:22 GMT):
@guoger
guoger (Sat, 09 Feb 2019 00:11:44 GMT):
- those params are put there to be changed (to mimic realistic scenario or whatever we want to test)
- i'm mostly interested in blocks per second w/p batching
yacovm (Sat, 09 Feb 2019 00:13:56 GMT):
I don't think that blocks *number* per second is a metric we need to increase.... we need to increase the block *size* per second
guoger (Sat, 09 Feb 2019 00:16:35 GMT):
yeah... ultimately bits per second :)
guoger (Sat, 09 Feb 2019 00:21:00 GMT):
and of course the benchmark params and measurements are malleable. i think it's another way to aid memory leak debug. (did a rudimentary experiment by putting periodic `runtime.ReadMemStats` to a background goroutine in orderer, and i don't see an obvious memory leak...)
guoger (Sat, 09 Feb 2019 00:21:06 GMT):
will continue later today
guoger (Sat, 09 Feb 2019 15:45:15 GMT):
Screen Shot 2019-02-09 at 23.42.13.png
guoger (Sat, 09 Feb 2019 15:45:21 GMT):
```
const (
EnvSize = 2000 // size of envelope payload in bytes
ClientPerNode = 5 // number of clients connecting to each orderer
TxPerClient = 20000 // number of tx sent by each client
ClusterSize = 3 // number of orderers in cluster
SnapshotInterval = "10 MB" // take snapshto if cumulative data exceeds limit
MaxMessageCount = 50 // MaxMessageCount in BatchSize
)
```
```
Type: etcdraft Nodes: 3 Broadcast Clients: 15 Total Tx: 300000 Elapsed Time: 87.53s Write tps: 3427.4 tx/s
```
guoger (Sat, 09 Feb 2019 15:45:45 GMT):
uploaded a plot of memory footprint
guoger (Sat, 09 Feb 2019 15:45:45 GMT):
uploaded a plot of memory footprint (memory sample rate: 100ms)
guoger (Sat, 09 Feb 2019 15:45:45 GMT):
uploaded a plot of memory footprint (memory sample rate: 100ms, gathered using in-proc `runtime.ReadMemStats`)
guoger (Sat, 09 Feb 2019 15:45:45 GMT):
uploaded a plot of memory footprint (memory sample rate: 100ms, gathered using in-proc `runtime.ReadMemStats`) Avg mem usage: ~30MB
yacovm (Sat, 09 Feb 2019 16:40:11 GMT):
It'll be interesting to find the maximum point of the throughput, as a function of the max message count, where the other variables are fixed @guoger
yacovm (Sat, 09 Feb 2019 16:41:50 GMT):
It'll be interesting to find the absolute maximum of the throughput, as a function of the max message count, where the other variables are fixed @guoger
yacovm (Sat, 09 Feb 2019 16:42:43 GMT):
However, we must not forget that this is within the same machine.
tock (Sun, 10 Feb 2019 11:09:35 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=Zsyhegx5ZLah4BFCG) @guoger when they do not check, we generally drop the offending transaction, certainly in the broadcast phase (before Kafka ordering).
guoger (Sun, 10 Feb 2019 15:01:00 GMT):
a second thought - doesn't `consensus_etcdraft_committed_block_number ` overlap with `ledger.block_processing_time.%{channel} `?
guoger (Sun, 10 Feb 2019 15:01:00 GMT):
a second thought - doesn't `consensus_etcdraft_committed_block_number ` overlap with `ledger.block_processing_time.%{channel} `? cc @adarshsaraf123
adarshsaraf123 (Mon, 11 Feb 2019 09:04:33 GMT):
@guoger actually `ledger_blockchain_height` is closer. However, I am not sure if we end up exposing that metric too for raft nodes through ledger. I am trying to verify the same.
adarshsaraf123 (Mon, 11 Feb 2019 09:18:08 GMT):
From what I could make out, and I might be wrong, we do not expose the `ledger_blockchain_height` metric for orderer nodes and therefore retaining `consensus_etcdraft_committed_block_number` should be good.
guoger (Mon, 11 Feb 2019 13:39:25 GMT):
looking into FAB-14129
guoger (Mon, 11 Feb 2019 15:09:02 GMT):
@yacovm to continue the conversation during scrum. _In concept_ (regardless of how implementation looks right now), if a admin supplies a _trusted_ config block to start a channel, this channel should be able to start performing consensus based on that config block (verify hash links), while pulling blocks prior to that config block in background. Is this correct? (of course, ledger impl needs to be changed to allow committing a future block with gaps, assuming that block is trusted)
yacovm (Mon, 11 Feb 2019 15:15:24 GMT):
but the admin only uses the last config block from the system channel
yacovm (Mon, 11 Feb 2019 15:15:28 GMT):
not for all channels
yacovm (Mon, 11 Feb 2019 15:15:36 GMT):
you don't want to get 100 blocks if you have 100 channels
yacovm (Mon, 11 Feb 2019 15:15:43 GMT):
you want the OSN to do its own thing autonomously
guoger (Mon, 11 Feb 2019 15:21:20 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=xWWt5464hNsqn86Nb) @yacovm and how does this prevent us from doing what was described?
yacovm (Mon, 11 Feb 2019 15:27:40 GMT):
i still don't understand what you're proposing
yacovm (Mon, 11 Feb 2019 15:27:49 GMT):
are you proposing to supply last config blocks for all channels?
guoger (Mon, 11 Feb 2019 15:31:10 GMT):
no.. not really proposing anything :), just asking a question. I'm trying to understand the fundamental reason of why starting chain has to be dependent on pulling all missing blocks prior to supplied config block.
yacovm (Mon, 11 Feb 2019 15:34:28 GMT):
there are several reasons
yacovm (Mon, 11 Feb 2019 15:34:37 GMT):
1) because if you start the chain you need to commit the new blocks, but you can't because the ledger is in-order append
yacovm (Mon, 11 Feb 2019 15:35:23 GMT):
2) because if you start the chain, you can't validate the last config block of that chain before pulling the blocks before that
yacovm (Mon, 11 Feb 2019 15:36:46 GMT):
3) because if you start the chain and a peer/client asks from you a block you don't have - you can't service the request, and that's sort of a diversion from the existing error handling model
guoger (Mon, 11 Feb 2019 15:45:20 GMT):
1) yes of course, unless our ledger implementation allows committing of future blocks, assuming they are supplied by admin of _this_ orderer
2) what if that config block is given in the same way as genesis block, so it should be trusted?
3) that's true, although we could still have this orderer service the channel after all missing blocks are pulled, even though it participates in rafting already (just not seen from peer/client)
If i have a chance to rephrase my question, i could've said: is there any security concern or impossibility in theory, that prevents us from doing XYZ?
guoger (Mon, 11 Feb 2019 15:45:20 GMT):
1) yes of course, unless our ledger implementation allows committing of future blocks, assuming they are supplied by admin of _this_ orderer
2) what if that config block is given in the same way as genesis block, so it should be trusted?
3) that's true, although we could still have this orderer service the channel after all missing blocks are pulled, even though it participates in rafting already (just not seen from peer/client)
If i have a chance to rephrase my question, i could've said: is there any security concern or impossibility in theory, that prevents us from doing XYZ? @yacovm
yacovm (Mon, 11 Feb 2019 15:49:11 GMT):
if you have a cluster of 3 nodes and you slowly replace all nodes completely
yacovm (Mon, 11 Feb 2019 15:49:26 GMT):
you then need to ensure they replicated the blocks in all channels
yacovm (Mon, 11 Feb 2019 15:49:31 GMT):
otherwise - you have no data loss
yacovm (Mon, 11 Feb 2019 15:50:04 GMT):
with the current onboarding protocol - there is no way of making the user do that, i.e - you can never replace them all because they need to pull the blocks beforehand.
guoger (Mon, 11 Feb 2019 15:57:24 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=ie5vbFsRgfhbZzmeZ) @yacovm I guess you meant data loss?
yacovm (Mon, 11 Feb 2019 15:58:46 GMT):
that's what i said no?
yacovm (Mon, 11 Feb 2019 15:58:47 GMT):
data loss
guoger (Mon, 11 Feb 2019 16:00:44 GMT):
no data loss == data _not_ lost?
guoger (Mon, 11 Feb 2019 16:00:45 GMT):
shouldn't it be
> otherwise - you have data loss
?
yacovm (Mon, 11 Feb 2019 16:02:28 GMT):
yeah
guoger (Mon, 11 Feb 2019 16:03:10 GMT):
just to make i get to this correct, you need to make sure there are at least 1 node in system channel that has full data to prevent data loss. Although pulling blocks in parallel with starting the chain _violates_ this, for the reason you described
guoger (Mon, 11 Feb 2019 16:03:10 GMT):
@yacovm just to make i get to this correct, you need to make sure there are at least 1 node in system channel that has full data to prevent data loss. Although pulling blocks in parallel with starting the chain _violates_ this, for the reason you described
guoger (Mon, 11 Feb 2019 16:03:10 GMT):
@yacovm just to make i get to this correct, you need to make sure there are at least 1 node in system channel that has full data at any given point of time, to prevent data loss. Although pulling blocks in parallel with starting the chain _violates_ this, for the reason you described
guoger (Mon, 11 Feb 2019 16:03:10 GMT):
@yacovm just to make i get to this correct, you need to make sure there are at least 1 node in system channel that has full data _at any given point of time_, to prevent data loss. Although pulling blocks in parallel with starting the chain _violates_ this, for the reason you described
yacovm (Mon, 11 Feb 2019 16:15:02 GMT):
not only 1 node
yacovm (Mon, 11 Feb 2019 16:15:16 GMT):
the number of nodes that you want to guard against data loss
yacovm (Mon, 11 Feb 2019 16:15:24 GMT):
for example if you have 3 nodes then you need at least 2 nodes, right?
yacovm (Mon, 11 Feb 2019 16:15:40 GMT):
because if only 1 node has the ledger, and that node is forever gone - you've lost the state
yacovm (Mon, 11 Feb 2019 16:15:43 GMT):
well, the ledger
yacovm (Mon, 11 Feb 2019 16:15:59 GMT):
that's why I am advocating against starting chains prematurely.
guoger (Mon, 11 Feb 2019 17:09:25 GMT):
gotcha, thanks for explaining this :)
guoger (Mon, 11 Feb 2019 23:37:06 GMT):
@yacovm what is your suspect in memory leak?
yacovm (Mon, 11 Feb 2019 23:39:06 GMT):
I suspect protobuf and gRPC
yacovm (Mon, 11 Feb 2019 23:41:27 GMT):
I uploaded a docker image to dockerhub (my private repo) that does 2 things:
1) in the step method - it makes [this](https://github.com/hyperledger/fabric/blob/release-1.4/orderer/consensus/etcdraft/chain.go#L363) a concrete type and not a pointer
yacovm (Mon, 11 Feb 2019 23:41:44 GMT):
2) In the cluster method - it nils out the payload after the dispatching is done
yacovm (Mon, 11 Feb 2019 23:41:44 GMT):
2) In the [Consensus](https://github.com/hyperledger/fabric/blob/master/orderer/common/cluster/comm.go#L115-L120) method - it nils out the payload after the dispatching is done
yacovm (Mon, 11 Feb 2019 23:44:26 GMT):
my wild shot in the dark guess is - that maybe gRPC somehow caches message objects, and the memory is still pointing to these buffer slices.
yacovm (Mon, 11 Feb 2019 23:53:51 GMT):
but the memory graphs [only reach](https://jira.hyperledger.org/secure/attachment/16793/02112019-top-command-output-on-orderers.txt) 500MB, i'd wish they reach 1GB or so.
guoger (Tue, 12 Feb 2019 02:06:17 GMT):
hmmm... but how come it's not observed in benchmark test...
C0rWin (Tue, 12 Feb 2019 08:19:07 GMT):
does QA able to reproduce memory leak running Raft OSNs? @yacovm @guoger @scottz
C0rWin (Tue, 12 Feb 2019 08:19:07 GMT):
does SVT able to reproduce memory leak running Raft OSNs? @yacovm @guoger @scottz
scottz (Tue, 12 Feb 2019 08:19:07 GMT):
Has joined the channel.
guoger (Tue, 12 Feb 2019 10:04:35 GMT):
I think they do?
scottz (Tue, 12 Feb 2019 13:24:25 GMT):
@C0rWin yes. We attached a new set of files yesterday to FAB-14054
guoger (Wed, 13 Feb 2019 10:05:13 GMT):
something i realized today (it's new to me, pls ignore if you already knew) - this does not properly recover panic:
```
func catch() {
if e := recover(); e != nil {
fmt.Printf("Panicking: %s\n", e)
}
}
func main() {
defer func() {
catch()
}()
panic("Blah")
}
```
because `recover` has to be called *directly* by a deferred func. This leads to [this snippet of code](https://github.com/hyperledger/fabric/blob/a7dddec829743edcf05c8b7f316859395e1cde40/integration/e2e/etcdraft_reconfig_test.go#L543-L546) not to take effect.
kostas (Wed, 13 Feb 2019 12:32:13 GMT):
I actually did not know that.
kostas (Wed, 13 Feb 2019 12:32:32 GMT):
That's good to know.
tock (Wed, 13 Feb 2019 20:50:32 GMT):
Here is a UT failure I haven't seen before: https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/8887/console
`15:12:07 FAIL github.com/hyperledger/fabric/orderer/common/cluster 24.053s`
@yacovm any ideas?
tock (Wed, 13 Feb 2019 20:50:32 GMT):
Here is a UT failure I haven't seen before: https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/8887/console
`15:12:07 FAIL github.com/hyperledger/fabric/orderer/common/cluster 24.053s`
@yacovm any ideas?
tock (Wed, 13 Feb 2019 20:50:32 GMT):
Here is a UT failure I haven't seen before: https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/8887/console
`15:12:07 FAIL github.com/hyperledger/fabric/orderer/common/cluster 24.053s`
@yacovm
tock (Wed, 13 Feb 2019 20:50:32 GMT):
Here is a UT failure I haven't seen before: https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/8887/console
`15:12:07 FAIL github.com/hyperledger/fabric/orderer/common/cluster 24.053s`
tock (Wed, 13 Feb 2019 20:51:44 GMT):
@yacovm any ideas?
yacovm (Wed, 13 Feb 2019 21:05:32 GMT):
there is a JIRA for that
yacovm (Wed, 13 Feb 2019 21:05:40 GMT):
I am aware...
tock (Wed, 13 Feb 2019 21:25:17 GMT):
sleepless in Haifa
adarshsaraf123 (Thu, 14 Feb 2019 06:34:16 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=hEPjLw2Bah4kqziQp) @guoger that's interesting! thanks :)
dexhunter (Thu, 14 Feb 2019 07:57:32 GMT):
Has joined the channel.
guoger (Thu, 14 Feb 2019 13:35:16 GMT):
@yacovm did you say that there's chance of OS thread leak? but doesn't `GOMAXPROCS` guards against it?
yacovm (Thu, 14 Feb 2019 13:35:41 GMT):
I think it's `-1` by default
guoger (Thu, 14 Feb 2019 13:36:07 GMT):
then it's the number of processors
guoger (Thu, 14 Feb 2019 13:36:07 GMT):
~then it's the number of processors~
guoger (Thu, 14 Feb 2019 13:37:29 GMT):
wait. it's number of processors be default. i don't know if it's possible to set it to -1...
guoger (Thu, 14 Feb 2019 14:02:04 GMT):
@yacovm @C0rWin @kostas when you guys have some cycles, pls take a look at https://gerrit.hyperledger.org/r/c/29236/3. We need to get it merged so that svt team can run all other tests with this change. If FAB-14129 happens again, we'll have more info. thx
guoger (Thu, 14 Feb 2019 14:02:04 GMT):
@yacovm @C0rWin @kostas when you guys have some cycles, pls take a look at https://gerrit.hyperledger.org/r/c/29236/3. We need to get it merged so that svt team can run all other tests with this change. then If FAB-14129 happens again, we'll have more info. thx
guoger (Thu, 14 Feb 2019 14:02:47 GMT):
it's a small CR that adds some more debug logs
yacovm (Thu, 14 Feb 2019 14:02:52 GMT):
why can't you just do `block.Header.Number` instead?
yacovm (Thu, 14 Feb 2019 14:03:09 GMT):
when you print `bc.logger.Debugf("Created block %d", bc.number)`
guoger (Thu, 14 Feb 2019 14:04:07 GMT):
aren't they the same?
yacovm (Thu, 14 Feb 2019 14:04:36 GMT):
i guess
yacovm (Thu, 14 Feb 2019 14:05:03 GMT):
ok +2ed
guoger (Thu, 14 Feb 2019 14:08:04 GMT):
any breaking news from Matt w.r.t. mem leak? :P
guoger (Fri, 15 Feb 2019 05:58:23 GMT):
@yacovm @kostas is it ok to leave debug turned on in etcdraf UT? so that if CI captures a flaky test, we can have more info, instead of trying to reproduce it locally with debug on? (i don't think this will pollute CI logs if test doesn't fail)
yacovm (Fri, 15 Feb 2019 07:51:56 GMT):
dont see why not
kostas (Fri, 15 Feb 2019 13:21:37 GMT):
Same. I think that's fine.
adarshsaraf123 (Sat, 16 Feb 2019 04:44:16 GMT):
@C0rWin @guoger @kostas @yacovm Have pushed an update on the metrics CR: https://gerrit.hyperledger.org/r/#/c/29153/
adarshsaraf123 (Sat, 16 Feb 2019 04:44:47 GMT):
It will be good to have it merged before the month long tests for raft-svt begins.
guoger (Sat, 16 Feb 2019 09:23:33 GMT):
+1'ed
adarshsaraf123 (Sat, 16 Feb 2019 09:37:17 GMT):
Thanks for the review!
kostas (Sat, 16 Feb 2019 16:24:53 GMT):
Looks great, I just reviewed it as well.
kostas (Sat, 16 Feb 2019 16:24:59 GMT):
CI is killing us though.
kostas (Sat, 16 Feb 2019 16:24:59 GMT):
CI and our tests are a bad combo though.
kostas (Sat, 16 Feb 2019 16:24:59 GMT):
Test failures in CI are killing us though.
kostas (Sat, 16 Feb 2019 16:25:20 GMT):
> EndToEnd reconfiguration and onboarding when the orderer certificates are all rotated [It] is still possible to onboard new orderers
kostas (Sat, 16 Feb 2019 16:25:20 GMT):
In Adarsh's CR https://gerrit.hyperledger.org/r/#/c/29153/
> EndToEnd reconfiguration and onboarding when the orderer certificates are all rotated [It] is still possible to onboard new orderers
kostas (Sat, 16 Feb 2019 16:26:36 GMT):
In Matt's CR, 3 consecutive failures https://gerrit.hyperledger.org/r/c/29054/#message-67296a40_0a896dac
kostas (Sat, 16 Feb 2019 16:26:36 GMT):
In Matt's CR, 3 consecutive failures: https://gerrit.hyperledger.org/r/c/29054/#message-67296a40_0a896dac
adarshsaraf123 (Sat, 16 Feb 2019 16:36:23 GMT):
Thanks for the reviews @kostas and @yacovm !
tock (Sun, 17 Feb 2019 13:52:09 GMT):
[Fail] Chain Multiple Raft nodes when 3/3 nodes are running when Snapshotting is enabled [It] lagged node can catch up using snapshot
https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/9039/console
FYI @guoger @yacovm
tock (Sun, 17 Feb 2019 13:52:09 GMT):
[Fail] Chain Multiple Raft nodes when 3/3 nodes are running when Snapshotting is enabled [It] lagged node can catch up using snapshot
https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/9039/console
UnitTest failure in CI, FYI @guoger @yacovm
guoger (Mon, 18 Feb 2019 15:15:38 GMT):
as I mentioned during scrum, we cannot rely on `support.Height()` to get the last block number while chain is running, because commit of block is done [async](https://github.com/hyperledger/fabric/blob/d3318c21dbaaa086c9836b331f6101a0ea2228ce/orderer/common/multichannel/blockwriter.go#L152-L155). More specifically, if we make a decision based on `support.Height()`, i.e. check if last block is config block, we may be actually checking the second last block. Instead, we should initialize a var `nextBlock` with `support.Height` (or something equivalent) when chain is started, and keep track of block height independently onwards. (this is what https://gerrit.hyperledger.org/r/c/29363/ does).
However, we are exposed to another risk, where `support.Block(nextBlock-1)` may return nil because that block is still being committed...
Ultimately it would be nice to solve this internally within `support`, i.e. guarantee that *if* `Height()` and `WriteBlock` are called from same go routine, `Height` reliably returns correct last block number. Although i think it requires significant rework of implementation of ledger reader/writer.
What we need to solve now, is how to handle `nil` returned by `support.Block(nextBlock-1)`. (retry? wait?) ideas? cc @yacovm @kostas @jyellick
guoger (Mon, 18 Feb 2019 15:15:38 GMT):
as I mentioned during scrum, we cannot rely on `support.Height()` to get the last block number while chain is running, because commit of block is done [async](https://github.com/hyperledger/fabric/blob/d3318c21dbaaa086c9836b331f6101a0ea2228ce/orderer/common/multichannel/blockwriter.go#L152-L155). More specifically, if we make a decision based on `support.Height()`, i.e. check if last block is config block, we may be actually checking the second last block. Instead, we should initialize a var `nextBlock` with `support.Height()` (or something equivalent) when chain is started, and keep track of block height independently onwards. (this is what https://gerrit.hyperledger.org/r/c/29363/ does).
However, we are exposed to another risk, where `support.Block(nextBlock-1)` may return nil because that block is still being committed...
Ultimately it would be nice to solve this internally within `support`, i.e. guarantee that *if* `Height()` and `WriteBlock` are called from same go routine, `Height` reliably returns correct last block number. Although i think it requires significant rework of implementation of ledger reader/writer.
What we need to solve now, is how to handle `nil` returned by `support.Block(nextBlock-1)`. (retry? wait?) ideas? cc @yacovm @kostas @jyellick
guoger (Mon, 18 Feb 2019 15:15:38 GMT):
as I mentioned during scrum, we cannot rely on `support.Height()` to get the last block number while chain is running, because commit of block is done [async](https://github.com/hyperledger/fabric/blob/d3318c21dbaaa086c9836b331f6101a0ea2228ce/orderer/common/multichannel/blockwriter.go#L152-L155). More specifically, if we make a decision based on `support.Height()`, i.e. check if last block is config block, we may be actually checking the second last block. Instead, we should initialize a var `nextBlock` with `support.Height()` (or something equivalent) when chain is started, and keep track of block height independently onwards. (this is what https://gerrit.hyperledger.org/r/c/29363/ does).
However, we are exposed to another risk, where `support.Block(nextBlock-1)` may return nil because that block is still being committed...
Ultimately it would be nice to solve this internally within `support`, i.e. guarantee that *if* `Height()` and `WriteBlock` are called from same go routine, `Height` reliably returns correct last block number. Although i think it requires significant rework of implementation of ledger reader/writer.
What we need to solve now, is how to handle `nil` returned by `support.Block(nextBlock-1)`. (retry? wait?) ideas? cc @yacovm @kostas @jyellick
yacovm (Mon, 18 Feb 2019 16:24:19 GMT):
I dont understand what exactly is the risk with the nextblock-1?
guoger (Mon, 18 Feb 2019 23:20:03 GMT):
basically you are writing block i while reading it, which is racy @yacovm [ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=260b8e65-ea7c-47f9-950a-8930467ebade)
yacovm (Mon, 18 Feb 2019 23:36:27 GMT):
so just poll until it's not nil
kostas (Tue, 19 Feb 2019 01:10:29 GMT):
Yeah, not the prettiest solution but it'll do the trick.
guoger (Tue, 19 Feb 2019 07:40:04 GMT):
thinking a bit more, and i'd propose to keep a reference to `lastBlock` in chain, and use that whenever we do `support.Block(support.Height()-1)`, (except for chain.Start, where we still need to initialize `lastBlock` from support).
btw, I think FAB-14129 *is* also caused by this - after leader failover, new leader initializes a new `BlockCreator` with `support.Height()`, *which is 1 block behind*. Therefore, this new leader would end up proposing a block with the same number again.
guoger (Tue, 19 Feb 2019 07:40:04 GMT):
thinking a bit more, and i'd propose to keep a reference to `lastBlock` in chain, and use that whenever we do `support.Block(support.Height()-1)`, (except for chain.Start, where we still need to initialize `lastBlock` from support).
But for the time being, we can simply add retry, which is not pretty but simple enough
btw, I think FAB-14129 *is* also caused by this - after leader failover, new leader initializes a new `BlockCreator` with `support.Height()`, *which is 1 block behind*. Therefore, this new leader would end up proposing a block with the same number again.
guoger (Tue, 19 Feb 2019 07:40:34 GMT):
unfortunately we don't know this for sure because log was not full, and we don't if there was actually a leader failover before panic
guoger (Tue, 19 Feb 2019 07:40:53 GMT):
but this seems to be the most reasonable explanation to me
guoger (Tue, 19 Feb 2019 07:43:55 GMT):
as stated by svt team:
> The original problem was twice seen in a lab with slower disk read/writes (2 IOPS per GB),
so block commit was actually slow, and the chance of this is higher
yacovm (Tue, 19 Feb 2019 08:26:11 GMT):
I see
yacovm (Tue, 19 Feb 2019 08:26:20 GMT):
so the leader, gets two blocks from the Ready()
yacovm (Tue, 19 Feb 2019 08:26:27 GMT):
the real block, and the one it proposed
yacovm (Tue, 19 Feb 2019 08:27:03 GMT):
@guoger - that sounds to me, like we need to ensure the leader "saw" the tip of the replicated log, *before* it initialzied the block creator
yacovm (Tue, 19 Feb 2019 08:28:49 GMT):
*OR* what we could do, is just have all OSNs:
1) Poll the block height after commit to ensure it has been committed
2) When receiving a block from the Ready() channel and its number has been seen before - grab all transactions from it, and schedule them to be cut once again into some next block
yacovm (Tue, 19 Feb 2019 08:28:56 GMT):
@guoger what do you say?
guoger (Tue, 19 Feb 2019 08:30:08 GMT):
solution is actually simple - don't do `support.Block(support.Height()-1)`
guoger (Tue, 19 Feb 2019 08:30:44 GMT):
instead, keep a reference to the last block
guoger (Tue, 19 Feb 2019 08:31:20 GMT):
(we have all the knowledge in chain, and we don't need to read them from ledger)
guoger (Tue, 19 Feb 2019 08:31:54 GMT):
@yacovm
yacovm (Tue, 19 Feb 2019 08:34:26 GMT):
but what if you are a fresh new leader, and you still haven't seen the last block proposed by the previous leader from the `Ready` channel, and you cut a new block now with the same sequence?
guoger (Tue, 19 Feb 2019 08:36:19 GMT):
we already have the logic to wait for block inflight before start accepting new tx (and only create BlockCreator by then)
guoger (Tue, 19 Feb 2019 08:36:30 GMT):
when a node is elected as new leader
guoger (Tue, 19 Feb 2019 08:37:08 GMT):
it's not the race there. it's the race between `blockwriter.commitBlock` and `support.Height`
yacovm (Tue, 19 Feb 2019 08:37:23 GMT):
hmmm ok
yacovm (Tue, 19 Feb 2019 08:37:26 GMT):
if you say so
jyellick (Tue, 19 Feb 2019 14:37:35 GMT):
@tock https://gerrit.hyperledger.org/r/c/28439/25/integration/nwo/configtx_template.go#88
jyellick (Tue, 19 Feb 2019 14:38:12 GMT):
Why do we not want to enable the V2_0 orderer capability in all tests? We expect for standard deployments to do this.
tock (Tue, 19 Feb 2019 15:14:10 GMT):
I think we need it to be configurable. We will probably want to write tests that verify what happens when this capability is off. Regardless, when I started migration, I did not want to alter the setup of any other test but mine, and this is the original reason I did it this way. Every test that wants to enable this can do something like this:
```
conf := nwo.BasicKafka()
conf.OrdererCap.V2_0 = true
network = nwo.New(conf, testDir, client, BasePort(), components)
```
(In light of our discussion here https://jira.hyperledger.org/browse/FAB-14180, I'll have to change this to V1_4_1, and add the same support for channel capabilities)).
tock (Tue, 19 Feb 2019 15:14:10 GMT):
I think we need it to be configurable. We will probably want to write tests that verify what happens when this capability is off. Regardless, when I started migration, I did not want to alter the setup of any other test but mine, and this is the original reason I did it this way. Every test that wants to enable this can do something like this:
```
conf := nwo.BasicKafka()
conf.OrdererCap.V2_0 = true
network = nwo.New(conf, testDir, client, BasePort(), components)
```
(In light of our discussion here https://jira.hyperledger.org/browse/FAB-14180, I'll have to change this to V1_4_1, and add the same support for channel capabilities)).
jyellick (Tue, 19 Feb 2019 15:16:25 GMT):
I think we need to be a wary of trying to make our integration test suite do all things. And, as a rule we should default to using the latest capabilities. You'll notice there is no control for any of the other capabilities in tests, and if there's no need to disable the V2_0 capability for any of the tests to work, I'd just as soon not have the option.
jyellick (Tue, 19 Feb 2019 15:16:25 GMT):
I think we need to be a wary of trying to make our integration test suite do all things. And, as a rule we should default to using the latest capabilities. You'll notice there is no control for any of the other capabilities in tests, and as there's no need to disable the V2_0 capability for any of the tests to work, I'd just as soon not have the option.
jyellick (Tue, 19 Feb 2019 15:17:07 GMT):
If we _really_ think we need be able to control which capabilities are enabled in the integration tests (which, I am not convinced of), we should default to the newest set of capabilities, not require them to be explicitly enabled for all tests.
jyellick (Tue, 19 Feb 2019 15:18:04 GMT):
There's the larger test suite around things like upgrade, which makes sense to live outside of the mainline integration tests.
jyellick (Tue, 19 Feb 2019 15:20:05 GMT):
(FYI, the reason I ran up against this is because I had updated the configtx yaml generation to include the new v20 capability and encountered a merge conflict, and couldn't figure out why there was the 'if' switch around it)
tock (Tue, 19 Feb 2019 15:23:13 GMT):
I am fine with policy. As I said, when I started I wanted to play safe, not knowing if this would break other tests. You convinced me ;-)
tock (Tue, 19 Feb 2019 15:23:13 GMT):
I am fine with this policy. As I said, when I started I wanted to play safe, not knowing if this would break other tests. You convinced me ;-)
kostas (Wed, 20 Feb 2019 14:21:12 GMT):
Looking for another review and +2 on this one: https://gerrit.hyperledger.org/r/c/29323/
Pradeep_Pentakota (Thu, 21 Feb 2019 04:01:25 GMT):
Has joined the channel.
guoger (Thu, 21 Feb 2019 06:51:25 GMT):
@yacovm @kostas i'm going to bump etcd/raft lib to include `23731bf`, in order to fix https://jira.hyperledger.org/browse/FAB-13920
guoger (Thu, 21 Feb 2019 06:51:25 GMT):
@yacovm @kostas i'm going to bump etcd/raft lib version to include `23731bf`, in order to fix https://jira.hyperledger.org/browse/FAB-13920
guoger (Thu, 21 Feb 2019 06:51:25 GMT):
@yacovm @kostas i'm going to bump etcd/raft lib version to include [23731bf](https://github.com/etcd-io/etcd/commit/23731bf9ba556867089a9cc8db5e492ca6035fe8), in order to fix https://jira.hyperledger.org/browse/FAB-13920
guoger (Thu, 21 Feb 2019 06:51:25 GMT):
@yacovm @kostas i'm going to bump etcd/raft lib version to include [23731bf](https://github.com/etcd-io/etcd/commit/23731bf9ba556867089a9cc8db5e492ca6035fe8), in order to fix https://jira.hyperledger.org/browse/FAB-13920. I've confirmed that etcd is affected by the same issue w/o this commit (long leader failover when `PreVote` and `CheckQuorum` are both true). Also, we get to pass logger to wal & snap pkg with version being bumped
guoger (Thu, 21 Feb 2019 09:48:16 GMT):
@yacovm can we maybe reduce the rate of printing this log (if we need to print it at all):
```
[e][OrdererOrg.orderer1] 2019-02-21 17:46:04.202 CST [orderer.consensus.etcdraft] consensusSent -> DEBU 1ca1 Sending msg of 29 bytes to 2 on channel systemchannel took 1.535µs
```
kostas (Fri, 22 Feb 2019 14:48:42 GMT):
Before Monday's scrum, can you have a look at the placeholder story for backlogged items and see if there is anything that _should_ make the cut for 1.4.1? I am basically looking for items that are backlogged when though they shouldn't be.
kostas (Fri, 22 Feb 2019 14:48:42 GMT):
Before Monday's scrum, can you have a look at the placeholder story for backlogged items and see if there is anything that _should_ make the cut for 1.4.1? I am basically looking for items that are backlogged when they shouldn't be.
jyellick (Fri, 22 Feb 2019 15:56:17 GMT):
Hey guys ( @kostas @guoger @yacovm @tock ). I've been having some discussions with @braduf who is working with a group of banks, building a consortium around HLF. He and some of his colleagues are familiar with golang and would like to get involved in consensus development starting with Raft. For now, while they're getting up to speed, he's interested in contributing to documentation and possibly test.
braduf (Fri, 22 Feb 2019 15:56:17 GMT):
Has joined the channel.
yacovm (Fri, 22 Feb 2019 15:57:46 GMT):
sure... the more people testing, the better
yacovm (Fri, 22 Feb 2019 15:58:09 GMT):
as for documentation I am not sure what exactly he has in mind?
braduf (Fri, 22 Feb 2019 16:02:52 GMT):
Hi all, thank you @jyellick for the introduction! Well, since we first should get familliar with the work you guys already did, we thought we could write some documentation in our learning process and when we have a better understanding help with coding and testing tasks that are pending. But of course, it's up to you to tell us what needs to be done
kostas (Fri, 22 Feb 2019 16:04:02 GMT):
@braduf: I think documentation as you pick things up is a great entry point. See: https://jira.hyperledger.org/browse/FAB-12892
yacovm (Fri, 22 Feb 2019 16:04:07 GMT):
just running it in your environment would help a ton
yacovm (Fri, 22 Feb 2019 16:05:49 GMT):
and yes like Kostas said - deployment guides are very much welcome. If you can deploy it, and then write a guide on how to deploy it in *some environment* - we can review it and add it to the official documentation.
yacovm (Fri, 22 Feb 2019 16:06:22 GMT):
@braduf what environment do you guys have, how do you plan to run Fabric?
yacovm (Fri, 22 Feb 2019 16:06:55 GMT):
Kubernetes? Bare metal? docker containers without orchestration? OpenStack? private cloud, etc?
braduf (Fri, 22 Feb 2019 16:07:21 GMT):
we are planning to run it with Kubernetes and in AWS
yacovm (Fri, 22 Feb 2019 16:07:34 GMT):
awesome
braduf (Fri, 22 Feb 2019 16:08:08 GMT):
from our side, the other entities will all have the cloud provider of their choice, but we will start in AWS with kubernetes
braduf (Fri, 22 Feb 2019 16:12:02 GMT):
ok, so from what i understood is that what is on the master branch is what we should build?
yacovm (Fri, 22 Feb 2019 16:13:32 GMT):
yes...
guoger (Fri, 22 Feb 2019 16:35:02 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=yXzesS2wnHcAYJTf6) @kostas as i'm going through them, should their fix version be marked to v1.4.1?
kostas (Fri, 22 Feb 2019 16:35:17 GMT):
Sure.
guoger (Fri, 22 Feb 2019 16:47:51 GMT):
marked 5 tasks (out of open & not assigned): FAB-13655 FAB-14075 FAB-13654 FAB-13722
guoger (Fri, 22 Feb 2019 16:48:31 GMT):
they are not critical, but something _nice to have_
guoger (Fri, 22 Feb 2019 16:50:10 GMT):
also, if there's a raft-related integration test failure, i'd appreciate to be alerted (either pinged in jira with link to build log or here) if that's not too much to ask, thx!
C0rWin (Sun, 24 Feb 2019 12:21:45 GMT):
Hey guys,
Considering porting staff into 1.4.0, I've create a JIRA to keep track and pulled all Raft related changes from master since cut of 1.4.0.
Please see updated list in https://jira.hyperledger.org/browse/FAB-14321 and comment in case I've missed something.
To create the list I've used following set of commands:
```
alias gr='ssh -p 29418 PUT_HERE_YOUR_USERNAME@gerrit.hyperledger.org gerrit'
```
then you should be able to run queries like this:
```
gr query 'status:merged (owner:"Jay Guo
C0rWin (Sun, 24 Feb 2019 12:21:45 GMT):
Hey guys,
Considering porting staff into 1.4.0, I've create a JIRA to keep track and pulled all Raft related changes from master since cut of 1.4.0.
Please see updated list in https://jira.hyperledger.org/browse/FAB-14321 and comment in case I've missed something.
To create the list I've used following set of commands:
```
alias gr='ssh -p 29418 PUT_HERE_YOUR_USERNAME@gerrit.hyperledger.org gerrit'
```
then you should be able to run queries like this:
```
gr query 'status:merged (owner:"Jay Guo
C0rWin (Sun, 24 Feb 2019 12:41:22 GMT):
and query could be pipelined actually to extract only relevant information like this:
```
gr query 'status:merged (owner:"Jay Guo
C0rWin (Sun, 24 Feb 2019 12:41:22 GMT):
and query could be pipelined actually to extract only relevant information like this:
```
gr query 'status:merged (owner:"Jay Guo
C0rWin (Sun, 24 Feb 2019 12:41:22 GMT):
and query could be pipelined actually to extract only relevant information like this:
```
gr query 'status:merged (owner:"Jay Guo
C0rWin (Sun, 24 Feb 2019 12:49:55 GMT):
items in JIRA listed from earliest to latest so it will be easier to keep track
yacovm (Sun, 24 Feb 2019 17:19:51 GMT):
it seems that there are non raft changes in there
yacovm (Sun, 24 Feb 2019 17:20:28 GMT):
such as https://gerrit.hyperledger.org/r/#/c/28305/
yacovm (Sun, 24 Feb 2019 17:21:31 GMT):
and https://gerrit.hyperledger.org/r/#/c/29068/
yacovm (Sun, 24 Feb 2019 17:22:34 GMT):
or https://gerrit.hyperledger.org/r/#/c/29087/ . Is there a way to pass to the gerrit query a file path of changes? :thinking:
yacovm (Sun, 24 Feb 2019 17:23:01 GMT):
i.e - change set contains a file in `./orderer`
kostas (Sun, 24 Feb 2019 18:06:16 GMT):
@C0rWin: Thanks for the first pass! I'm interested in whether what Yacov is saying is feasible. Failing that, I think we'll need to do a second pass manually grepping the commits since 1.4 was cut and figuring out if anything is missing from your list.
kostas (Sun, 24 Feb 2019 18:06:49 GMT):
For instance, Jason's configxten changes (stack starting from: https://gerrit.hyperledger.org/r/c/29143/) plus his fix on the orderer protos (https://gerrit.hyperledger.org/r/c/29384/).
C0rWin (Sun, 24 Feb 2019 21:20:04 GMT):
@kostas Well, manual pass over list is something we cannot escape, I will take a look on Jason stack
C0rWin (Sun, 24 Feb 2019 21:21:26 GMT):
> or https://gerrit.hyperledger.org/r/#/c/29087/ . Is there a way to pass to the gerrit query a file path of changes? 🤔
yes, gerrit supports such queries
C0rWin (Sun, 24 Feb 2019 21:24:21 GMT):
I've took out the list 3 items you mentioned, overlooked them while was inspecting resulted set, thanks for pointing out @yacovm
guoger (Mon, 25 Feb 2019 03:46:34 GMT):
@C0rWin thanks for putting this together! i actually don't know you can run query against gerrit. I see you've tagged us for some reviews, although I don't know what to do with it exactly?
C0rWin (Mon, 25 Feb 2019 08:23:04 GMT):
@guoger those items is to backport Raft from master into v1.4.1 branch
guoger (Mon, 25 Feb 2019 08:24:42 GMT):
@C0rWin I know but I'm not familiar with backporting process :( i suppose they are just waiting for 2 +2 to be merged, right?
C0rWin (Mon, 25 Feb 2019 08:32:32 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=Qui3DR4XkNqKct4a8) @guoger basically, yes, but the key here is to go over the list of CRs from master I've posted above and see whenever I've overlooked something or there is something we can skip
C0rWin (Mon, 25 Feb 2019 08:32:57 GMT):
I mean, I'm only cherry-picking CRs + resolving conflicts
C0rWin (Mon, 25 Feb 2019 08:33:16 GMT):
no code change done (or at least not suppose to be done)
guoger (Mon, 25 Feb 2019 08:34:08 GMT):
ah, got it. thx!
tock (Tue, 26 Feb 2019 08:52:02 GMT):
The fix for https://jira.hyperledger.org/browse/FAB-14327 is in: https://gerrit.hyperledger.org/r/#/c/29535/
I think this is important for 1.4.1, please take a look @kostas @C0rWin @yacovm
tock (Tue, 26 Feb 2019 08:52:02 GMT):
The fix for https://jira.hyperledger.org/browse/FAB-14327 is in: https://gerrit.hyperledger.org/r/#/c/29535/
I think this is important for 1.4.1, please take a look @kostas @C0rWin @yacovm @jyellick
tock (Tue, 26 Feb 2019 08:53:47 GMT):
The BYFN sample with a 3-node etcd/raft cluster is also pending review: https://gerrit.hyperledger.org/r/#/c/29506/
yacovm (Tue, 26 Feb 2019 08:56:14 GMT):
I think the bug fix should be cherry picked all the way to v1.1
yacovm (Tue, 26 Feb 2019 09:25:32 GMT):
or to all releases which have been introduced with the optimization of synchronicity is
yacovm (Tue, 26 Feb 2019 09:25:32 GMT):
or to all releases which have been introduced with the optimization of synchronicity
C0rWin (Tue, 26 Feb 2019 10:58:02 GMT):
> or to all releases which have been introduced with the optimization of synchronicity
IMO it was there since v1.1
tock (Tue, 26 Feb 2019 14:21:00 GMT):
These two CRs add channel capability V1_4_1, gate migration with it in the Bundle, and clean the Bundle from processing related to migration. I think they should make it to v1.4.1, so that we won't have to update the peers beyond 1.4.1 when we introduce and enable migration in the orderers. Please take a look @kostas @C0rWin @yacovm @jyellick
https://gerrit.hyperledger.org/r/#/c/29585/
https://gerrit.hyperledger.org/r/#/c/29586/
C0rWin (Thu, 28 Feb 2019 08:57:58 GMT):
Hi guys
C0rWin (Thu, 28 Feb 2019 08:58:26 GMT):
the work of porting Raft into v1.4.1 finished you can find CRs cherry picked here: https://gerrit.hyperledger.org/r/#/q/owner:bartem%2540il.ibm.com+status:open
C0rWin (Thu, 28 Feb 2019 08:58:44 GMT):
test flakiness a bit killing us
C0rWin (Thu, 28 Feb 2019 08:59:16 GMT):
however I'd like to encourage you to review the list and verify nothing is missing
C0rWin (Thu, 28 Feb 2019 08:59:36 GMT):
I admit that I might have missed a few items from past couple of days
C0rWin (Thu, 28 Feb 2019 08:59:57 GMT):
@kostas @yacovm @guoger @tock @adarshsaraf123 ^^^^
C0rWin (Thu, 28 Feb 2019 09:00:22 GMT):
This is the JIRA to track status of items being merged: https://jira.hyperledger.org/browse/FAB-14321
guoger (Thu, 28 Feb 2019 09:17:44 GMT):
thanks for all the effort! this is an exhausting work....
C0rWin (Thu, 28 Feb 2019 09:22:20 GMT):
@guoger tell me :joy: still fighting flaky tests
tock (Thu, 28 Feb 2019 10:59:09 GMT):
looks good to me, as far as the migration commits go...
however see https://gerrit.hyperledger.org/r/#/c/29535/ for the blockwriter issue
tock (Thu, 28 Feb 2019 10:59:23 GMT):
not strictly raft...
guoger (Thu, 28 Feb 2019 13:44:54 GMT):
https://jira.hyperledger.org/browse/FAB-14380 racy back-to-back configs
https://jira.hyperledger.org/browse/FAB-14413 newly onboarded node is leader, lagged node is not able to catch up
guoger (Thu, 28 Feb 2019 13:46:28 GMT):
@yacovm saw [your comment](https://jira.hyperledger.org/browse/FAB-14413?focusedCommentId=57722&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-57722) there, willing to hear your solution :)
yacovm (Thu, 28 Feb 2019 13:46:38 GMT):
so let's before that just consider something else
yacovm (Thu, 28 Feb 2019 13:46:47 GMT):
at least - talk about it
yacovm (Thu, 28 Feb 2019 13:47:06 GMT):
if we have such a situation, we have a node that is behind, and at least a quorum of nodes alive, right?
kostas (Thu, 28 Feb 2019 13:47:57 GMT):
Correct.
yacovm (Thu, 28 Feb 2019 13:48:01 GMT):
in theory - if we have an operations team and they forcefully kill the leader - then there is a takeover :joy:
yacovm (Thu, 28 Feb 2019 13:48:12 GMT):
but I don't think this is a very.... elegant solution
kostas (Thu, 28 Feb 2019 13:48:17 GMT):
Also correct.
yacovm (Thu, 28 Feb 2019 13:48:23 GMT):
but i am just saying - it's a last resort, in a sense
kostas (Thu, 28 Feb 2019 13:49:16 GMT):
So as we noted in the call earlier, this happens because o4 is supposed to reach to the leader for the Raft snapshot? @guoger
yacovm (Thu, 28 Feb 2019 13:49:23 GMT):
as for my solution, I can just extend my logic to make blocks being pulled if there is no leader for a long time regardless of the node is evicted or not
kostas (Thu, 28 Feb 2019 13:50:53 GMT):
Right! That's what I'm building towards, I think.
yacovm (Thu, 28 Feb 2019 13:51:38 GMT):
Jay is right that if there is a leader change in the meantime then we need to worry about double commit
guoger (Thu, 28 Feb 2019 13:51:41 GMT):
i think that's what i was suggesting in that jira as well? and i was saying that we should update the chain with the block number, so that it does not replay the same block
kostas (Thu, 28 Feb 2019 13:51:41 GMT):
@yacovm Can you go over how exactly your logic reacts _now_, i.e. given the current code
yacovm (Thu, 28 Feb 2019 13:52:12 GMT):
right now - if it detects its eviction, it just first halts the chain and then pulls blocks until the last block
guoger (Thu, 28 Feb 2019 13:52:14 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=L2RFt7y2aqDCs5Jbq) @yacovm i actually thought about the same thing lol
yacovm (Thu, 28 Feb 2019 13:52:42 GMT):
we need some mechanism to suspend its commit while this is done
kostas (Thu, 28 Feb 2019 13:52:54 GMT):
> i think that's what i was suggesting in that jira as well? and i was saying that we should update the chain with the block number, so that it does not replay the same block
Sorry I missed the most recent comments there.
kostas (Thu, 28 Feb 2019 13:52:54 GMT):
> i think that's what i was suggesting in that jira as well? and i was saying that we should update the chain with the block number, so that it does not replay the same block
Sorry ab out that -- I missed the most recent comments there. I see it now.
yacovm (Thu, 28 Feb 2019 13:53:17 GMT):
no, it won't be enough
yacovm (Thu, 28 Feb 2019 13:53:26 GMT):
we have 2 goroutines that might commit
kostas (Thu, 28 Feb 2019 13:54:09 GMT):
Again: can we go over how the block pulling logic works now?
guoger (Thu, 28 Feb 2019 13:54:49 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=FhLCAnzxDQB32FZGB) @yacovm exactly, that's why i think we should have only one go routine doing commits, to avoid race
kostas (Thu, 28 Feb 2019 13:55:05 GMT):
In this particular case, what prevents us from pulling blocks from other nodes in the consenter set is the fact that the channel is leaderless?
kostas (Thu, 28 Feb 2019 13:55:18 GMT):
(According to o4's view as least.)
yacovm (Thu, 28 Feb 2019 13:55:30 GMT):
the leader is the only one sending you snapshots :(
guoger (Thu, 28 Feb 2019 13:55:43 GMT):
block pulling is always pulling committed blocks, so it's orthogonal to leader state
yacovm (Thu, 28 Feb 2019 13:56:04 GMT):
Clipboard - February 28, 2019 3:56 PM
yacovm (Thu, 28 Feb 2019 13:56:16 GMT):
http://blog.carlosgaldino.com/a-brief-overview-of-the-raft-algorithm.html
guoger (Thu, 28 Feb 2019 13:56:16 GMT):
(and to be accurate, it's not really about snapshots. this situation is general enough w/o snapshotting)
guoger (Thu, 28 Feb 2019 13:56:16 GMT):
(and to be accurate, it's not really about snapshots. this situation is general enough w/o snapshotting. This problem can happen even if snapshotting is disabled)
kostas (Thu, 28 Feb 2019 13:56:52 GMT):
Ah, so based on the snippet above, an attempt to send a snapshot happens automatically.
kostas (Thu, 28 Feb 2019 13:57:03 GMT):
Whether the follower node is fairly caught up or not.
kostas (Thu, 28 Feb 2019 13:58:11 GMT):
That actually answers my question then -- I was wondering if there was something in the block pulling logic that makes o4 stop early. (And if that is the case, we fix that so that o4 is fairly caught up and we avoid the need for snapshotting.) But if I'm reading the snippet above correctly, this is useless.
guoger (Thu, 28 Feb 2019 13:58:20 GMT):
i think solution is actually straightforward, we just send an artificial signal to `snapC` to trigger a catchup
yacovm (Thu, 28 Feb 2019 13:58:54 GMT):
if there is no leader for a long time?
kostas (Thu, 28 Feb 2019 13:59:06 GMT):
Before we go over that:
guoger (Thu, 28 Feb 2019 13:59:07 GMT):
yep
yacovm (Thu, 28 Feb 2019 13:59:29 GMT):
I like it @guoger
kostas (Thu, 28 Feb 2019 13:59:34 GMT):
I'm not sure I follow the double-commit comment.
yacovm (Thu, 28 Feb 2019 13:59:40 GMT):
so the double commit...
yacovm (Thu, 28 Feb 2019 14:00:14 GMT):
imagine we have the goroutine that performs `chain.apply()` because there was a leader change
yacovm (Thu, 28 Feb 2019 14:00:23 GMT):
but at the same time - we pull blocks and commit them from the other goroutine
kostas (Thu, 28 Feb 2019 14:01:15 GMT):
Understood.
kostas (Thu, 28 Feb 2019 14:01:51 GMT):
What is the condition under which you'll send the artifical signal to `snapC` @guoger?
kostas (Thu, 28 Feb 2019 14:01:51 GMT):
What is the condition under which you'll send the artificial signal to `snapC` @guoger?
guoger (Thu, 28 Feb 2019 14:02:21 GMT):
how often is EvictionSuspicion performed?
kostas (Thu, 28 Feb 2019 14:02:48 GMT):
Yacov was supposed to push a CR that bumps the timing from every second to 10 seconds or so?
yacovm (Thu, 28 Feb 2019 14:03:04 GMT):
@guoger in `evictionSuspector.confirmSuspicion` in `chain.go` we can just add a dirty bit toggle to the `chain.go` and it can then check for it, and then artificially trigger a snapshot on itself
yacovm (Thu, 28 Feb 2019 14:03:12 GMT):
I can push it now, Kostas (the 10 seconds)
yacovm (Thu, 28 Feb 2019 14:03:18 GMT):
want me to ?
guoger (Thu, 28 Feb 2019 14:04:00 GMT):
why not passing `snapC` to suspector?
kostas (Thu, 28 Feb 2019 14:04:14 GMT):
There is no rush. Just letting Jay now that the answer today won't be the same as the answer tomorrow, or in a few days from now.
guoger (Thu, 28 Feb 2019 14:04:59 GMT):
> Just letting Jay now that the answer today won't be the same as the answer tomorrow, or in a few days from now.
not following this...
kostas (Thu, 28 Feb 2019 14:05:17 GMT):
Eviction suspicion is performed every second now.
kostas (Thu, 28 Feb 2019 14:05:29 GMT):
But we'll adjust that frequency.
guoger (Thu, 28 Feb 2019 14:06:46 GMT):
my suggestion:
- pass `snapC` to `suspector`
- `suspector` knows latest config block, and knows it contains its own cert in consenter set
- `suspector` triggers an artificial snap signal
guoger (Thu, 28 Feb 2019 14:06:46 GMT):
my suggestion:
- pass `snapC` to `suspector`
- `suspector` knows latest config block, and knows it contains its own cert in consenter set
- `suspector` triggers an artificial snap signal at certain frequency
kostas (Thu, 28 Feb 2019 14:07:29 GMT):
I am following this.
yacovm (Thu, 28 Feb 2019 14:07:31 GMT):
> why not passing `snapC` to suspector?
We can do that... sure
yacovm (Thu, 28 Feb 2019 14:07:51 GMT):
let me push a draft @kostas @guoger ok?
guoger (Thu, 28 Feb 2019 14:07:58 GMT):
sure
kostas (Thu, 28 Feb 2019 14:08:00 GMT):
Perfect, thanks.
kostas (Thu, 28 Feb 2019 14:08:51 GMT):
This is actually a great find. Thanks J.
guoger (Thu, 28 Feb 2019 14:08:56 GMT):
@C0rWin wanna have a quick chat on
https://jira.hyperledger.org/browse/FAB-14380 racy back-to-back configs ?
guoger (Thu, 28 Feb 2019 14:09:18 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=HsqzXttAJLxAA7o57) @kostas haha, i'd like to thank @yacovm 's great integration test
yacovm (Thu, 28 Feb 2019 14:09:38 GMT):
the integration test actually doesn't make sense
yacovm (Thu, 28 Feb 2019 14:09:43 GMT):
but i still added it
guoger (Thu, 28 Feb 2019 14:09:45 GMT):
otherwise i wouldn't be able to find it
yacovm (Thu, 28 Feb 2019 14:09:49 GMT):
out of paranoid because that use case wasn't covered
guoger (Thu, 28 Feb 2019 14:09:49 GMT):
great gut feeling
yacovm (Thu, 28 Feb 2019 14:09:58 GMT):
no, it's parania
yacovm (Thu, 28 Feb 2019 14:09:58 GMT):
no, it's paranoia
yacovm (Thu, 28 Feb 2019 14:10:02 GMT):
not gut feeling
guoger (Thu, 28 Feb 2019 14:10:18 GMT):
lol
yacovm (Thu, 28 Feb 2019 14:10:43 GMT):
but great find because everyone else ignored the occasional failure
yacovm (Thu, 28 Feb 2019 14:11:22 GMT):
if an integration test fails in the forest and someone re-verifies, did it really fail? :thinking:
guoger (Thu, 28 Feb 2019 14:14:52 GMT):
just updated the description there
guoger (Thu, 28 Feb 2019 14:15:43 GMT):
since i've dealing with test flakes for a while, got this habit to skim through failure logs already :joy:
guoger (Thu, 28 Feb 2019 14:15:43 GMT):
since i've been dealing with test flakes for a while, got this habit to skim through failure logs already :joy:
yacovm (Thu, 28 Feb 2019 14:18:02 GMT):
@guoger - how do i explicitly ask for a snapshot without knowing the `raftpb.Snapshot` ?
guoger (Thu, 28 Feb 2019 14:23:09 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=C8MjeiEFgJKspCXGn) @yacovm what you could do is to manually assemble one, and leave the `Metadata` in it unset. and in `serveReq` where it's handled, you can differentiate _actual snapshot_ and _artificial snapshot_ by inspecting `Metadata.Index`. If it's non-zero, it's actual one, otherwise it's artificial
guoger (Thu, 28 Feb 2019 14:27:58 GMT):
and make sure to add a explicit check [here](https://github.com/hyperledger/fabric/blob/f97797659fe0f7c11ec53ac7af40ad3c29898430/orderer/consensus/etcdraft/chain.go#L934)
```
if block.Header.Number != c.lastBlock.Header.Number+1 {
break
}
```
guoger (Thu, 28 Feb 2019 14:29:49 GMT):
because raft has *no* knowledge about this block pulling in the background, and will still continue from where it's left, once communication is "repaired". And all missing entries are still sent via normal raft protocal
guoger (Thu, 28 Feb 2019 14:29:50 GMT):
and i suppose you'll need to call `configureComm` once that last config block is committed
yacovm (Thu, 28 Feb 2019 14:30:49 GMT):
hmmm
yacovm (Thu, 28 Feb 2019 14:31:04 GMT):
you raise a point here - do we call configureComm in `catchUp` ?
yacovm (Thu, 28 Feb 2019 14:32:20 GMT):
@guoger
C0rWin (Thu, 28 Feb 2019 14:33:49 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=fePiaRYQTsBEXpYC2) @guoger sorry, I'm on call... can we chat a bit later?
guoger (Thu, 28 Feb 2019 14:36:58 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=DykbKMGPsL3AbRZsQ) @yacovm and i think we always had this problem :joy: if it's not expensive, we could call it in `catchUp` when we write a config block (it's idempotent, right?)
guoger (Thu, 28 Feb 2019 14:36:58 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=DykbKMGPsL3AbRZsQ) @yacovm and i think this is another problem we always had :joy: if it's not expensive, we could call it in `catchUp` when we write a config block (it's idempotent, right?)
yacovm (Thu, 28 Feb 2019 14:37:35 GMT):
oh god
kostas (Thu, 28 Feb 2019 14:38:43 GMT):
I added a few comments to FAB-14380. I do not _think_ the last proposal is correct.
guoger (Thu, 28 Feb 2019 14:41:50 GMT):
@kostas replied. I didn't mean to change `ProcessConfigMsg`, but just to perform additional check after we've done with `ProcessConfigMsg`, in chain.go
kostas (Thu, 28 Feb 2019 14:42:39 GMT):
> if it's not expensive, we could call it in `catchUp` when we write a config block
Is there a problem with this proposal? This sounds right to me.
guoger (Thu, 28 Feb 2019 14:46:41 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=gFuDiABbexS7S2edd) @kostas it just surfaces another problem buried in my code :( if a node is added/removed, and the block that adds/removes it is pulled via puller, then it's not able to communicate
guoger (Thu, 28 Feb 2019 14:46:41 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=gFuDiABbexS7S2edd) @kostas it just surfaces another problem buried in my code :( if a node is added/removed, and the block that adds/removes it is pulled via puller because of snapshotting, then it's not able to communicate
guoger (Thu, 28 Feb 2019 14:48:17 GMT):
will solve this altogether
dave.enyeart (Thu, 28 Feb 2019 15:08:17 GMT):
we are checking the codebase for proto maps since they don't get serialized deterministically... just wanted to be sure this one is ok: https://github.com/hyperledger/fabric/blob/master/protos/orderer/etcdraft/configuration.pb.go#L219
dave.enyeart (Thu, 28 Feb 2019 15:10:13 GMT):
another one here @jyellick : https://github.com/hyperledger/fabric/blob/master/protos/common/configuration.pb.go#L214
dave.enyeart (Thu, 28 Feb 2019 15:10:23 GMT):
just checking...
yacovm (Thu, 28 Feb 2019 15:13:28 GMT):
in raft, the leader cuts the blocks - so there should be only a single instance of that
kostas (Thu, 28 Feb 2019 15:50:07 GMT):
Yeah, that map in etcdraft/configuration.proto shouldn't be a problem.
jyellick (Thu, 28 Feb 2019 17:29:42 GMT):
@dave.enyeart Yes, the config proto stuff is deliberately only marshaled in one place. Everyone unmarshals and checks the updates for correctness, but they use the original marshaled form to commit, they do not remarshal it.
kostas (Fri, 01 Mar 2019 18:06:47 GMT):
Please have a look at @J's stack starting from: https://gerrit.hyperledger.org/r/c/29693/
kostas (Fri, 01 Mar 2019 18:06:47 GMT):
Please have a look at @guoger's stack starting from https://gerrit.hyperledger.org/r/c/29693/ and if it looks good to you too, we can get it in.
C0rWin (Sun, 03 Mar 2019 15:09:53 GMT):
```14:05:17 • Failure [11.203 seconds]
14:05:17 Chain
14:05:17 /w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain_test.go:65
14:05:17 Multiple Raft nodes
14:05:17 /w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain_test.go:1143
14:05:17 when reconfiguring raft cluster
14:05:17 /w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain_test.go:1274
14:05:17 reconfiguration
14:05:17 /w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain_test.go:1344
14:05:17 stop cluster quorum and continue reconfiguration after the restart [It]
14:05:17 /w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain_test.go:1585
14:05:17
14:05:17 Timed out after 10.000s.
14:05:17 Expected
14:05:17
C0rWin (Sun, 03 Mar 2019 15:10:17 GMT):
is someone aware of the failure above? is it a flake or a bug?
C0rWin (Sun, 03 Mar 2019 15:10:24 GMT):
found here: https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/9828/console
C0rWin (Sun, 03 Mar 2019 15:10:33 GMT):
related to https://gerrit.hyperledger.org/r/#/c/29645/
guoger (Mon, 04 Mar 2019 12:27:56 GMT):
Artem has seen [this failure](https://jenkins.hyperledger.org/job/fabric-verify-integration-tests-x86_64/6641/consoleFull) consistently. I _think_ the fix is simple - we should put `network.EventuallyTimeout` in
```
launch := func(o *nwo.Orderer) {
orderers = append(orderers, o)
runner := network.OrdererRunner(o)
ordererRunners = append(ordererRunners, runner)
process := ifrit.Invoke(grouper.Member{Name: o.ID(), Runner: runner})
Eventually(process.Ready()).Should(BeClosed()) // This should use network.EventuallyTimeout
ordererProcesses = append(ordererProcesses, process)
}
```
However, I don't know why master is not affected by this? (or i'm completely missing the point here?) @yacovm
yacovm (Mon, 04 Mar 2019 12:35:01 GMT):
I don't think so. It is how it is used in like 10 places in the code
yacovm (Mon, 04 Mar 2019 12:35:04 GMT):
in that test
yacovm (Mon, 04 Mar 2019 12:35:24 GMT):
if it is problematic then the test would've failed in like ~ 10 places
yacovm (Mon, 04 Mar 2019 12:36:09 GMT):
it shouldn't take more than a second to start an orderer process
yacovm (Mon, 04 Mar 2019 12:36:23 GMT):
the default timeout is a second
guoger (Mon, 04 Mar 2019 13:06:11 GMT):
hmmm.. there's ~7 sec delay...
```
19:37:21 [e][OrdererOrg.orderer1new] 2019-03-04 11:37:05.255 UTC [orderer.common.cluster.replication] fetchLastBlockSeq -> INFO 013 127.0.0.1:31006 is at block sequence of 1
19:37:21 [e][OrdererOrg.orderer1new] 2019-03-04 11:37:12.248 UTC [orderer.common.cluster.replication] probeEndpoint -> WARN 014 Failed connecting to 127.0.0.1:31009: failed to create new connection: context deadline exceeded
```
guoger (Mon, 04 Mar 2019 13:22:30 GMT):
by comparing it to the log of successful run, i'm expecting
```
Failed connecting to 127.0.0.1:36009: failed to create new connection: connection error: desc = "transport: error while dialing: dial tcp 127.0.0.1:36009: connect: connection refused"
```
it should be `connection refused`, but not `context deadline exceeded`, which is configured to 7 sec IIRC.
guoger (Mon, 04 Mar 2019 13:23:00 GMT):
what's your take on this @yacovm ?
yacovm (Mon, 04 Mar 2019 13:24:13 GMT):
i don't understand
guoger (Mon, 04 Mar 2019 13:49:37 GMT):
to summarize so that others can have context, what that integration test does is:
1. create a cluster with <1, 2, 3, 4>
2. start <1, 2, 3> only
3. add a new node <5>
4. start <5>
5. wait for it to sync with network
6. kill <1, 2, 3>
7. start <4>
8. waiting for <4> to start suspecting eviction
9. resurrect <1, 2, 3>
10. everything should keep working properly, eventually
So at step 4, when <5> is started, it probes other nodes in the channel, <1, 2, 3, 4>. Connections to <1, 2, 3> are fine, although <4> is not started, hence should *refuse* connection. However, in the failed integration test, I see `context deadline exceeded`
guoger (Mon, 04 Mar 2019 13:49:37 GMT):
to summarize so that others can have context, what that integration test does is:
1. create a cluster with <1, 2, 3, 4>
2. start <1, 2, 3> only
3. add a new node <5>
4. start <5>
5. wait for it to sync with network
6. kill <1, 2, 3>
7. start <4>
8. waiting for <4> to start suspecting eviction
9. resurrect <1, 2, 3>
10. everything should keep working properly, eventually
So at step 4, when <5> is started, it probes other nodes in the channel, <1, 2, 3, 4>. Connections to <1, 2, 3> are fine, although <4> is not started, hence should *refuse* connection. However, in the failed integration test, I see `context deadline exceeded`, which is 7 sec delay and causes test to fail
guoger (Mon, 04 Mar 2019 13:49:56 GMT):
@yacovm
yacovm (Mon, 04 Mar 2019 13:53:58 GMT):
ah you're saying that the 7 seconds delay at onboarding causes the test to fail?
guoger (Mon, 04 Mar 2019 13:55:52 GMT):
i don't think there's anything wrong with onboarding though... why doesn't <4>, a never started process, refuses connection? that's my question...
yacovm (Mon, 04 Mar 2019 13:56:17 GMT):
i don't know... can you link me to a CI log?
yacovm (Mon, 04 Mar 2019 13:56:21 GMT):
a console output in CI
guoger (Mon, 04 Mar 2019 13:56:36 GMT):
https://jenkins.hyperledger.org/job/fabric-verify-integration-tests-x86_64/6641/consoleFull
guoger (Mon, 04 Mar 2019 13:56:59 GMT):
this is part of backporting work
guoger (Mon, 04 Mar 2019 13:57:29 GMT):
maybe some CR is missing, which prevents probe from failing fast?
yacovm (Mon, 04 Mar 2019 14:11:19 GMT):
I don't understand why you think the 7 seconds is bad.
yacovm (Mon, 04 Mar 2019 14:11:29 GMT):
OK there is a 7 seconds delay, but why is that a problem?
yacovm (Mon, 04 Mar 2019 14:11:36 GMT):
the failure is not due to this
yacovm (Mon, 04 Mar 2019 14:12:07 GMT):
it's due to the process not starting fast enough:
```
13:40:47 Timed out after 1.000s.
13:40:47 Expected
13:40:47 <<-chan struct {} | len:0, cap:0>: 0xc0017b5920
```
guoger (Mon, 04 Mar 2019 14:12:33 GMT):
i'm not saying 7 sec timeout is bad.... trying to figure why that test fails
yacovm (Mon, 04 Mar 2019 14:12:43 GMT):
ah ok
yacovm (Mon, 04 Mar 2019 14:12:52 GMT):
so i think the test fails because the OSN hasn't started in time
yacovm (Mon, 04 Mar 2019 14:12:57 GMT):
the process, that is
guoger (Mon, 04 Mar 2019 14:13:03 GMT):
and the process does not start fast enough because it got `context deadline exceeded` while probing <4>
guoger (Mon, 04 Mar 2019 14:13:23 GMT):
even though <4> is not started. and i expect it to be `connection refused` though
yacovm (Mon, 04 Mar 2019 14:13:23 GMT):
i don't think that's it... why do you think it's related?
guoger (Mon, 04 Mar 2019 14:14:02 GMT):
because logs suggests 7 sec delay in starting orderer, which is apparently greater than 1 sec
yacovm (Mon, 04 Mar 2019 14:14:23 GMT):
but the 1 second is for the process to start, no?
yacovm (Mon, 04 Mar 2019 14:14:52 GMT):
it's not like the `Ifrit` understands onboarding
yacovm (Mon, 04 Mar 2019 14:15:20 GMT):
:thinking_face:
guoger (Mon, 04 Mar 2019 14:15:44 GMT):
process fulfills `StartCheck` when it says `Beginning to serve requests`
guoger (Mon, 04 Mar 2019 14:15:54 GMT):
which is after the probing, no>?
yacovm (Mon, 04 Mar 2019 14:16:06 GMT):
oh i didn't know that
guoger (Mon, 04 Mar 2019 14:16:25 GMT):
```
config := ginkgomon.Config{
AnsiColorCode: n.nextColor(),
Name: o.ID(),
Command: cmd,
StartCheck: "Beginning to serve requests",
StartCheckTimeout: 15 * time.Second,
}
```
yacovm (Mon, 04 Mar 2019 14:16:45 GMT):
ok then that's the problem....
yacovm (Mon, 04 Mar 2019 14:16:52 GMT):
we shouldn't have this at all
yacovm (Mon, 04 Mar 2019 14:17:03 GMT):
it should be something else
yacovm (Mon, 04 Mar 2019 14:17:07 GMT):
but how does it not fail in master?
guoger (Mon, 04 Mar 2019 14:17:12 GMT):
exactly
guoger (Mon, 04 Mar 2019 14:17:50 GMT):
was there a CR that does something like `skip retry if connection is refused`?
yacovm (Mon, 04 Mar 2019 14:18:21 GMT):
there is, not not at startup ;)
yacovm (Mon, 04 Mar 2019 14:18:26 GMT):
only in dynamic onboarding
yacovm (Mon, 04 Mar 2019 14:18:56 GMT):
and the problem here is that you connect to 4 OSNs
yacovm (Mon, 04 Mar 2019 14:19:00 GMT):
but 1 of them is offline
yacovm (Mon, 04 Mar 2019 14:19:07 GMT):
so you can only return when the timeout expires
yacovm (Mon, 04 Mar 2019 14:19:16 GMT):
it's a scatter-gather pattern
yacovm (Mon, 04 Mar 2019 14:19:36 GMT):
but in any case we should not use `Beginning to serve requests` for that
guoger (Mon, 04 Mar 2019 14:22:39 GMT):
i think we are talking past each other... on `master`, `probe` to <4> fails and returns instantly with `connection refused`, however on 1.4.1 backporting, it waits for 7 sec timeout..
yacovm (Mon, 04 Mar 2019 14:32:09 GMT):
oh, i see.... this is odd. very odd
C0rWin (Mon, 04 Mar 2019 14:53:42 GMT):
the failure seems pretty consistent
C0rWin (Mon, 04 Mar 2019 14:53:45 GMT):
`14:40:45 [Fail] EndToEnd reconfiguration and onboarding when an orderer node is joined [It] isn't influenced by outdated orderers `
C0rWin (Mon, 04 Mar 2019 14:53:52 GMT):
https://jenkins.hyperledger.org/job/fabric-verify-integration-tests-x86_64/6678/console
C0rWin (Mon, 04 Mar 2019 14:54:21 GMT):
@yacovm @guoger can you take a look the commits stack to see if something if missing or committed out of natural order?
yacovm (Mon, 04 Mar 2019 14:54:49 GMT):
i will try
guoger (Mon, 04 Mar 2019 15:17:19 GMT):
i'll look into this tomorrow morning as well, in case yacov's schedule is packed. @C0rWin
yacovm (Mon, 04 Mar 2019 16:00:26 GMT):
I looked and seems like my CRs are all in master :/
yacovm (Mon, 04 Mar 2019 16:00:26 GMT):
I looked and seems like my CRs are all in v.1.4.1 :/
C0rWin (Mon, 04 Mar 2019 22:19:21 GMT):
https://gerrit.hyperledger.org/r/#/q/owner:bartem%2540il.ibm.com+status:open two last remaining CRs seems both of them hitting problem of integration tests described above
yacovm (Mon, 04 Mar 2019 23:46:42 GMT):
this is interesting... on both 1.4 and master branches, the OS returns that the port of the OSN is not listening. However - on v1.4, the gRPC client keeps retrying to dial until the timeout expires. On master- it fails fast. I ran wireshark, and on 1.4 there are 20 packets of SYN sent to the localhost, and on master there are only 7.... @mastersingh24 any idea why gRPC acts differently on v1.4 ?
I also printed the dial timeout and it is 7 seconds in both master and v1.4
yacovm (Mon, 04 Mar 2019 23:47:57 GMT):
specifically, like @guoger said:
on master we get the error:
```
[e][OrdererOrg.orderer1new] 2019-03-05 01:34:27.823 IST [orderer.common.cluster.replication] probeEndpoint -> WARN 026 Failed connecting to 127.0.0.1:36009: failed to create new connection: connection error: desc = "transport: error while dialing: dial tcp 127.0.0.1:36009: connect: connection refused"
```
while on v1.4 we get `probeEndpoint -> WARN 022 Failed connecting to 127.0.0.1:31009: failed to create new connection: context deadline exceeded`
yacovm (Tue, 05 Mar 2019 00:08:42 GMT):
@C0rWin me and @mastersingh24 solved it :grin:
yacovm (Tue, 05 Mar 2019 00:09:42 GMT):
there is a change set https://gerrit.hyperledger.org/r/#/c/29808/ that changes gRPC behavior slightly
yacovm (Tue, 05 Mar 2019 00:09:56 GMT):
and the integration test passes now for me on v1.4
yacovm (Tue, 05 Mar 2019 00:11:58 GMT):
Clipboard - March 5, 2019 2:11 AM
yacovm (Tue, 05 Mar 2019 00:14:06 GMT):
I placed the cherry picked CR beneath the rest of the stack and recursively rebased up to the top
guoger (Tue, 05 Mar 2019 01:23:25 GMT):
yayy
guoger (Tue, 05 Mar 2019 01:57:01 GMT):
How does an outdated orderer get to know the certs and endpoints of newly added nodes? i.e.
Initial state: <1, 2, 3, 4>
- start <1, 2, 3> *only*
- add <5>, start <5>
- add <6>, start <6>
- add <7>, start <7>
- kill <1, 2, 3>
- start <4>
Now we have 4/7 nodes running however the cluster is out of service. The best we can do here, is to start at least one of <1, 2, 3>
guoger (Tue, 05 Mar 2019 01:57:09 GMT):
@yacovm @kostas
jyellick (Tue, 05 Mar 2019 02:31:10 GMT):
@guoger Could we treat this as a late join and simply start him using the latest config block from the system channel?
guoger (Tue, 05 Mar 2019 02:44:58 GMT):
@jyellick I think it should work.. will try to write an integration test to confirm
jyellick (Tue, 05 Mar 2019 03:32:52 GMT):
@guoger @yacovm I've heard a few different places that "collocating peers and orderers is really bad", which, I understand that if you have two processes involved in a loop of work, that it's generally a bad idea to stick them together if you're at all interested in scale. But, is there some more severe reason why this is bad? Maybe something to do with fsync and block writes or something? Or am I just reading too much into this?
jyellick (Tue, 05 Mar 2019 03:32:52 GMT):
@guoger @yacovm I've heard a few different places that "collocating peers and orderers is really bad", which, I understand that if you have two processes involved in a loop of work, that it's generally a bad idea to stick them together if you're at all interested in scale. But, is there some more specifc reason why this is bad? Maybe something to do with fsync and block writes or something? Or am I just reading too much into this?
guoger (Tue, 05 Mar 2019 03:42:49 GMT):
Any reference? logically peer and orderer shouldn't interfere with each other when collocated. As for `fsync` in etcd/wal, there are some odd behavior on OSX, where kernel performs pretty bad at handling multiple `fsync` syscalls in parallel. Although I don't think fileledger actually uses the same syscall as etcd/wal (which may be a problem because bytes are not flushed, I've talked to Manish before and he perhaps will take a look there). But again, this is all on OSX
guoger (Tue, 05 Mar 2019 03:42:49 GMT):
Any reference? logically peer and orderer shouldn't interfere with each other when collocated IMO. As for `fsync` in etcd/wal, there are some odd behavior on OSX, where kernel performs pretty bad at handling multiple `fsync` syscalls in parallel. Although I don't think fileledger actually uses the same syscall as etcd/wal (which may be a problem because bytes are not flushed, I've talked to Manish before and he perhaps will take a look there). But again, this is all on OSX
jyellick (Tue, 05 Mar 2019 03:44:14 GMT):
It stuck out to me as Yacov's suspicion for why SVT has been seeing problems in their environment, then it was highlighted again at the beginning of Joe's draft document. Sounds like I am indeed reading more into this than there is.
guoger (Tue, 05 Mar 2019 03:48:31 GMT):
hmmm...
```
For Raft, peers and orderers should not be co-located in the same virtual machine.
```
I don't know how this conclusion is reached actually
jyellick (Tue, 05 Mar 2019 03:49:08 GMT):
It did seem like it was trying to call out some very particular issue to me.
jyellick (Tue, 05 Mar 2019 03:49:08 GMT):
It did seem like it was trying to call out some very particular issue to me. (Though ~For Raft~, peers and orderer should not be co-located in the same virtual machine reads reasonably to me)
jyellick (Tue, 05 Mar 2019 03:49:08 GMT):
It did seem like it was trying to call out some very particular issue to me. (Though "~For Raft~, peers and orderer should not be co-located in the same virtual machine reads reasonably to me")
jyellick (Tue, 05 Mar 2019 03:49:08 GMT):
It did seem like it was trying to call out some very particular issue to me. (Though:
~For Raft~, peers and orderer should not be co-located in the same virtual machine
reads reasonably to me)
guoger (Tue, 05 Mar 2019 03:54:53 GMT):
As far as raft is concerned, the performance bottleneck should be disk I/O (and that's by definition, because data must be persisted _before_ committed to state machine). therefore, all it matters is how fast can process write data entries to disk sequentially. However, I don't think this can cause contention between peer and orderer?
guoger (Tue, 05 Mar 2019 03:54:53 GMT):
As far as raft is concerned, the performance bottleneck should be disk I/O (and that's by definition, because data must be persisted _before_ committed to state machine). therefore, all it matters is how fast can a process write data entries to disk sequentially. However, I don't think this can cause contention between peer and orderer?
jyellick (Tue, 05 Mar 2019 04:01:30 GMT):
Makes sense. Okay, great, thanks for the insight @guoger !
yacovm (Tue, 05 Mar 2019 06:30:44 GMT):
@guoger look at the certificate rotation integration test.... It is what you describe
yacovm (Tue, 05 Mar 2019 06:30:45 GMT):
@guoger look at the certificate rotation integration test.... It is what you describe
yacovm (Tue, 05 Mar 2019 06:44:57 GMT):
@jyellick dont colocate because peers committing blocks to different channels in parallel and in independant times to the disk, will kill the WAL I/O
adarshsaraf123 (Tue, 05 Mar 2019 07:08:20 GMT):
What's the procedure now for new CRs? Do we implement on master and also cherry-pick to v1.4?
guoger (Tue, 05 Mar 2019 07:12:57 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=6855f791-4ada-4aff-b3b2-21ab079ef4d7) @yacovm this?
```
When("the orderer certificates are all rotated", func() {
It("is still possible to onboard new orderers", func() {
```
guoger (Tue, 05 Mar 2019 08:02:33 GMT):
@yacovm The test you referred to is not what i described. could you take a look here https://gerrit.hyperledger.org/r/c/29812/ ? I expect that integration test to pass, but it doesn't :(
sergefdrv (Tue, 05 Mar 2019 09:44:32 GMT):
Has left the channel.
yacovm (Tue, 05 Mar 2019 10:19:47 GMT):
no... it's nto that
yacovm (Tue, 05 Mar 2019 10:20:20 GMT):
```
When("the orderer certificates are all rotated", func() {
It("is still possible to onboard new orderers", func() {
```
yacovm (Tue, 05 Mar 2019 10:20:21 GMT):
@guoger
C0rWin (Tue, 05 Mar 2019 11:18:20 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=AhDKeabn9Ceev7hQu) @adarshsaraf123 yes
dave.enyeart (Tue, 05 Mar 2019 14:50:15 GMT):
@guoger I noticed some of your recently merged CRs are not in release-1.4... please cherry pick all your CRs to release-1.4.
dave.enyeart (Tue, 05 Mar 2019 14:50:15 GMT):
@guoger I noticed some of your recently merged CRs are not in release-1.4... please cherry pick all your Raft CRs to release-1.4.
guoger (Tue, 05 Mar 2019 14:51:16 GMT):
@dave.enyeart gotcha, doing it now. btw, i suppose we should wait for CRs on master to be merged before cherry-picking, is it correct?
dave.enyeart (Tue, 05 Mar 2019 14:53:07 GMT):
doesn't necessarily have to be merged yet... but yes it is better to wait until a CR goes through review cycles, otherwise you create more work for yourself and cause some confusion about which CR people should review
dave.enyeart (Tue, 05 Mar 2019 14:53:37 GMT):
typically we push initial CR to master, then cherry pick to release-1.4.
dave.enyeart (Tue, 05 Mar 2019 14:54:32 GMT):
however with the time pressure with raft on release v1.4.1, we may soon flip that approach - create CRs in release-1.4 then cherry pick to master. wait for an announcement from Jason on that...
dave.enyeart (Tue, 05 Mar 2019 14:55:00 GMT):
if you haven't cherry picked many... know that for most you can simply click Cherry Pick button in gerrit...that's the easiest way
yacovm (Tue, 05 Mar 2019 14:55:21 GMT):
wait wait hold on please... if we merge change sets into v1.4 in a different order than master, it will cause problems
dave.enyeart (Tue, 05 Mar 2019 14:56:16 GMT):
many times there are no dependencies right... in those cases order doesn't matter right
yacovm (Tue, 05 Mar 2019 14:57:19 GMT):
I think all changes these days are in the `chain.go` :)
adarshsaraf123 (Tue, 05 Mar 2019 15:12:10 GMT):
Okay, so the procedure as I understand would be to work for the CR on master (until Jason announces otherwise) and once it gets merged cherry-pick it to v1.4.1
jyellick (Tue, 05 Mar 2019 15:21:01 GMT):
Yep. Let's go ahead and submit to master for now, and cherry-pick to release-1.4 on merge.
jyellick (Tue, 05 Mar 2019 15:30:14 GMT):
@guoger https://gerrit.hyperledger.org/r/c/29735/ looks ready to merge and cherry-pick to release-1.4?
guoger (Tue, 05 Mar 2019 15:31:03 GMT):
@jyellick yep
jyellick (Tue, 05 Mar 2019 15:33:18 GMT):
Would you prefer to cherry-pick or would you rather I track these -- I'm fine either way
jyellick (Tue, 05 Mar 2019 15:33:18 GMT):
Would you prefer to cherry-pick or would you rather I track these? -- I'm fine either way
guoger (Tue, 05 Mar 2019 15:33:24 GMT):
@jyellick although shouldn't we first cherry-pick CRs merged recently prior to this? I'm going through gerrit to click the button now
guoger (Tue, 05 Mar 2019 15:33:27 GMT):
i can track it
jyellick (Tue, 05 Mar 2019 15:33:38 GMT):
Perfect
guoger (Tue, 05 Mar 2019 15:34:22 GMT):
but probably need some help at beginning... not familiar with cherry-picking process
jyellick (Tue, 05 Mar 2019 15:35:32 GMT):
Sure. You may either use the the gerrit UI, or, you may use git on the CLI. Personally I usually prefer to just use git, especially with stacked cherry-picks, but either should be fine
jyellick (Tue, 05 Mar 2019 15:36:14 GMT):
If you want to use the UI, you can simply click 'more' -> 'cherry-pick' and specify 'release-1.4' as the destination branch
jyellick (Tue, 05 Mar 2019 15:37:09 GMT):
If you'd rather use the more traditional git workflow, simply do a:
```git fetch origin && git checkout origin/release-1.4
git cherry-pick
jyellick (Tue, 05 Mar 2019 15:37:09 GMT):
If you'd rather use the more traditional git workflow, simply do a:
```git fetch origin && git checkout origin/release-1.4
git cherry-pick
jyellick (Tue, 05 Mar 2019 15:38:19 GMT):
If you use the gerrit UI, in order to achieve the stacking, we'll either need to cherry-pick and merge one at a time, or use the UI to rebase into the desired order. (Or someone can correct me if there's a better way through the UI)
jyellick (Tue, 05 Mar 2019 15:42:31 GMT):
Either way, let's coordinate here to try to get anything that's been merged into master and cherry-picked onto release-1.4 merged there asap
guoger (Tue, 05 Mar 2019 15:54:05 GMT):
question: should I cherry-pick Matt's protoutils changes, or revert my CRs to use old pkg on release-1.4?
yacovm (Tue, 05 Mar 2019 15:59:04 GMT):
i say - just change the v1.4 cherry pick to use the old sane utility code
yacovm (Tue, 05 Mar 2019 15:59:23 GMT):
before the protobuf move
jyellick (Tue, 05 Mar 2019 15:59:42 GMT):
+1 It seems simpler to use the old path than to attempt the move
yacovm (Tue, 05 Mar 2019 16:00:02 GMT):
it is also safer
yacovm (Tue, 05 Mar 2019 16:00:22 GMT):
now when i think about it.. who reviewed all these protobuf changes?
jyellick (Tue, 05 Mar 2019 16:01:20 GMT):
Gari/Matt were the 2+2 on most of them
dave.enyeart (Tue, 05 Mar 2019 16:22:34 GMT):
yeah, we decided not to backport the proto changes to limit the number of file changes in release-1.4... so the cherry picks that depend on the new proto package may need some manual updates
jyellick (Wed, 06 Mar 2019 05:29:09 GMT):
I've gone ahead and +2-ed what's passed CI in @guoger's cherry-pick stack as well as the ones in the stack leading up to 'repair corrupted WAL'. Obviously I'm still getting up to speed in this section of the code, so please do continue to review with rigor - I won't be at all offended to hear that I've +2-ed something dumb :slight_smile:
guoger (Wed, 06 Mar 2019 05:43:40 GMT):
Thanks!
adarshsaraf123 (Wed, 06 Mar 2019 06:41:03 GMT):
@C0rWin @jyellick Another bookkeeping question: are we to create separate JIRAs for master and v1.4 or do we just comment the gerrit urls for the CRs related to both on the same JIRA?
jyellick (Thu, 07 Mar 2019 04:48:42 GMT):
https://jira.hyperledger.org/browse/FAB-14499 @guoger @yacovm could one (or both) of you take a look at this one?
jyellick (Thu, 07 Mar 2019 04:50:30 GMT):
It might be helpful too for both our SVT and for someone like me trying to come up to speed to document the sorts of things we'd look for for underlying cause. I imagine overflowing queues for sends would be one. What about WAL writes, do we log or otherwise profile how long these are taking? Any other serviceability stuff we should be adding to assist in debug?
adarshsaraf123 (Thu, 07 Mar 2019 05:00:17 GMT):
@jyellick To partly answer your question on serviceability, I am currently working on adding two additional metrics: (link to JIRA: https://jira.hyperledger.org/browse/FAB-14473)
1. data_persist_duration: time taken for the etcdraft data to be persisted to storage (including the WAL writes)
2. # proposals received
I am almost done with the CR and am adding further tests before pushing the CR.
adarshsaraf123 (Thu, 07 Mar 2019 05:00:17 GMT):
@jyellick To partly answer your question on serviceability, I am currently working on adding two new etcdraft metrics: (link to JIRA: https://jira.hyperledger.org/browse/FAB-14473)
1. data_persist_duration: time taken for the etcdraft data to be persisted to storage (including the WAL writes)
2. # proposals received
I am almost done with the CR and am adding further tests before pushing the CR.
jyellick (Thu, 07 Mar 2019 05:10:27 GMT):
Ah, great! Thanks @adarshsaraf123 I imagine this would be very helpful in analyzing tests as we try to scale
guoger (Thu, 07 Mar 2019 06:07:45 GMT):
@jyellick also, as the code currently stands, if WAL sync takes more than 1 second, a warning is logged:
```
plog.Warningf("sync duration of %v, expected less than %v", took, warnSyncDuration)
```
This is from etcd/wal pkg and `warnSyncDuration` is hardcoded to 1 second
guoger (Thu, 07 Mar 2019 06:08:45 GMT):
we can sometimes observe this warning within SVT. But i think it doesn't occur that often after they switch to faster disk in IKS.
guoger (Thu, 07 Mar 2019 06:10:21 GMT):
@jyellick i took a look at the logs with DongMing from svt at FAB-14499 yesterday. Looks like OSN1 sends out heartbeat regularly, but those messages
guoger (Thu, 07 Mar 2019 06:10:21 GMT):
@jyellick i took a look at the logs with DongMing from svt at FAB-14499 yesterday. Looks like OSN1 sends out heartbeat regularly, but those messages do not reach receiver sometimes
guoger (Thu, 07 Mar 2019 06:16:05 GMT):
is it possible that this log doesn't necessarily indicate message being put on wire? @yacovm
```
orderer3rd-ordererorg 2019-02-28 03:21:46.215 UTC [orderer.common.cluster.step] sendMessage -> DEBU 3a4d89f Send of ConsensusRequest for channel testorgschannel139 with payload of size 28 to orderer2nd-ordererorg.ordererorg(orderer2nd-ordererorg:7050) took 10.316µs
```
Otherwise, either the message is super delayed, or receiver receives it but somehow failed to log it timely? I'm referring to this:
```
orderer3rd-ordererorg 2019-02-28 03:21:46.214 UTC [orderer.common.cluster.step] handleMessage -> DEBU 3a4d86f Received message from orderer2nd-ordererorg.ordererorg(172.30.181.167:43456): ConsensusRequest for channel testorgschannel139 with payload of size 28
```
guoger (Thu, 07 Mar 2019 06:16:29 GMT):
(ignore the timestamp. these two logs are just example)
yacovm (Thu, 07 Mar 2019 07:13:57 GMT):
Is that the 500 channels test Jay?
guoger (Thu, 07 Mar 2019 08:33:48 GMT):
@yacovm yes! the log is available in logDNA, which you should have access as well
yacovm (Thu, 07 Mar 2019 08:44:02 GMT):
then why do we even have debug logs? I said clearly we shouldn't have debug logs with 500 channels. it will kill the orderer's disk :(
yacovm (Thu, 07 Mar 2019 08:46:22 GMT):
https://github.com/hyperledger/fabric/blob/master/orderer/common/cluster/comm.go#L503-L524
yacovm (Thu, 07 Mar 2019 08:46:42 GMT):
> s it possible that this log doesn't necessarily indicate message being put on wire?
it indicates the message has been passed to the stream
yacovm (Thu, 07 Mar 2019 08:47:17 GMT):
> Otherwise, either the message is super delayed, or receiver receives it but somehow failed to log it timely? I'm referring to this:
Look for WARNings of the stream being aborted. it can happen
yacovm (Thu, 07 Mar 2019 08:47:17 GMT):
> Otherwise, either the message is super delayed, or receiver receives it but somehow failed to log it timely? I'm referring to this:
Look for WARNings of the stream being aborted. it can happen it sending a message takes more than 7 seconds. Then the stream is aborted
guoger (Thu, 07 Mar 2019 09:23:37 GMT):
@yacovm https://gerrit.hyperledger.org/r/c/29921/ here's the benchmark test that can be ran locally. Some parameters can be tuned
```
numOfOrd = 5
numOfChannels = 10
txPerChannelPerOrd = 200
txSize = 1000 // bytes
maxMessageCount = 1
```
(haven't tried more than 10 channels yet, worried about my laptop :P )
guoger (Thu, 07 Mar 2019 09:23:55 GMT):
But I do see leader failover sometimes
yacovm (Thu, 07 Mar 2019 09:25:28 GMT):
@guoger what is the throughput of the transactions?
guoger (Thu, 07 Mar 2019 09:27:36 GMT):
still wip, not measuring at the moment
guoger (Thu, 07 Mar 2019 09:30:57 GMT):
to create 500 channels though, i need to use another way other than `nwo.CreateChannel`, so they it doesn't wait for a channel to boot and can submit next config tx to sys channel.
guoger (Thu, 07 Mar 2019 09:31:19 GMT):
but for 10 channels, bootstrap time is still bearable
yacovm (Thu, 07 Mar 2019 09:31:27 GMT):
@guoger can you try your integration test with only 3 nodes?
yacovm (Thu, 07 Mar 2019 09:31:37 GMT):
wondering if there will be a failover then
guoger (Thu, 07 Mar 2019 09:33:35 GMT):
sure. I'm trying various combinations of parameters now. Looking at the params mentioned above, let me know if any particular combination you are interested in
guoger (Thu, 07 Mar 2019 09:34:12 GMT):
(also, when you have some cycles, you may also pull the CR and give it a try)
yacovm (Thu, 07 Mar 2019 09:35:08 GMT):
i will do it now
guoger (Thu, 07 Mar 2019 09:51:28 GMT):
updated it slightly to remove some noisy logs
guoger (Thu, 07 Mar 2019 09:52:09 GMT):
i created ~50G garbage on disk already... not sure why `AfterEach` is not ran sometimes...
yacovm (Thu, 07 Mar 2019 10:52:13 GMT):
Clipboard - March 7, 2019 12:52 PM
yacovm (Thu, 07 Mar 2019 10:52:22 GMT):
how am i supposed to run this, @guoger ?
yacovm (Thu, 07 Mar 2019 10:52:32 GMT):
and seems like the test only takes 14 seconds....
guoger (Thu, 07 Mar 2019 11:32:22 GMT):
I do ginkgo in perf dir, and you can increase those params. Default values are fairly small
yacovm (Thu, 07 Mar 2019 12:03:38 GMT):
oh i see
guoger (Thu, 07 Mar 2019 13:51:43 GMT):
@yacovm we cannot allow adding *and* removing node at the same time, simply because etcd/raft doesn't support it. (This is also suggested by raft thesis, even though altering 1+ nodes is feasible, but not recommended due to complexity added). IMO we could document the restriction right now, and support this later by allow in-place rotating -- if you had
yacovm (Thu, 07 Mar 2019 13:53:15 GMT):
this has nothing to do with the communication layer :joy:
yacovm (Thu, 07 Mar 2019 13:53:26 GMT):
a single node cluster
yacovm (Thu, 07 Mar 2019 13:53:32 GMT):
has nothing in the communication layer,
yacovm (Thu, 07 Mar 2019 13:53:45 GMT):
i am pretty sure that Artem's code skips yourself when you configure comm
guoger (Thu, 07 Mar 2019 13:53:53 GMT):
if we allow in-place cert swapping, it also applies to multi-node, no?
jyellick (Thu, 07 Mar 2019 13:54:00 GMT):
Another thought, which I'm not necessarily advocating, but in general, to rotate the TLS cert, I expect the node will need to be restarted. In a single node cluster, we could simply write a new config block to the chain before starting and the problem goes away.
yacovm (Thu, 07 Mar 2019 13:54:23 GMT):
are you saying to write a config block without going through consensus ?
jyellick (Thu, 07 Mar 2019 13:54:50 GMT):
In a single node network, it's arguable whether it's going through consensus or not
yacovm (Thu, 07 Mar 2019 13:55:05 GMT):
but what about the replicated log?
yacovm (Thu, 07 Mar 2019 13:55:33 GMT):
I guess it's not a problem though
jyellick (Thu, 07 Mar 2019 13:55:38 GMT):
I know less there, but on a single node it seems fixable
yacovm (Thu, 07 Mar 2019 13:55:38 GMT):
because the new node will onboard
yacovm (Thu, 07 Mar 2019 13:55:43 GMT):
hence pull all the chains
yacovm (Thu, 07 Mar 2019 13:56:08 GMT):
I like your idea, Jason.
jyellick (Thu, 07 Mar 2019 13:56:29 GMT):
Thanks, again to be clear, I'm not necessarily advocating this approach, but thought it might be worth considering.
yacovm (Thu, 07 Mar 2019 13:56:40 GMT):
@C0rWin what do you think?
guoger (Thu, 07 Mar 2019 13:58:15 GMT):
if we do in-place swap, we also make cert rotation simpler for multi-node, no?
yacovm (Thu, 07 Mar 2019 13:59:29 GMT):
what do you mean by in-place swap?
guoger (Thu, 07 Mar 2019 14:00:38 GMT):
all we want to do is to change a node to use cert_new, instead of cert_old, right? and this is stored in config block. (I may get this wrong and proposing a completely incorrect solution here)
guoger (Thu, 07 Mar 2019 14:01:16 GMT):
currently, diff
guoger (Thu, 07 Mar 2019 14:01:26 GMT):
i'm proposing to change this to [swap 1]
yacovm (Thu, 07 Mar 2019 14:02:04 GMT):
but you said we don't allow it in etcdraft
guoger (Thu, 07 Mar 2019 14:02:15 GMT):
it's agnostic to raft in this case
guoger (Thu, 07 Mar 2019 14:02:25 GMT):
we just reconfigure communication (for one node, it's no-op)
yacovm (Thu, 07 Mar 2019 14:02:30 GMT):
but we map IDs to certs using the metadata dont' we?
guoger (Thu, 07 Mar 2019 14:03:14 GMT):
yes, but why is that a problem?
yacovm (Thu, 07 Mar 2019 14:03:56 GMT):
so you are saying to just swap the metadata too
yacovm (Thu, 07 Mar 2019 14:04:13 GMT):
swap the pointer in the metadata that points C to 3 with a pointer that maps D to 3
guoger (Thu, 07 Mar 2019 14:04:30 GMT):
yes
yacovm (Thu, 07 Mar 2019 14:04:43 GMT):
the comm configuration is not the problem here, as it configures itself from scratch every time
guoger (Thu, 07 Mar 2019 14:04:52 GMT):
in the new config block. (it's consented anyway)
yacovm (Thu, 07 Mar 2019 14:04:54 GMT):
you can pass the same IDs, random IDs, it doesn't care
jyellick (Thu, 07 Mar 2019 14:04:58 GMT):
Seems like if we do not do 'swaps' we have another problem. In a 3 node network, in order to add from 3->4 we now need 3/4 nodes available for consensus. Which means that when we shut the 'doubled' node down, we lose consensus until he comes back up with the new cert. Hopefully a small window, maybe it's fine.
yacovm (Thu, 07 Mar 2019 14:05:00 GMT):
the problem here is the persistance in the metadata
yacovm (Thu, 07 Mar 2019 14:06:34 GMT):
@jyellick yes that is known - if you lose a node forever during reconfiguration.... this is a problem. But I guess you can always recover it by running 2 nodes - 1 with the old cert and 1 with the new.
guoger (Thu, 07 Mar 2019 14:07:20 GMT):
when we submit a config tx with
guoger (Thu, 07 Mar 2019 14:08:44 GMT):
if we create a an artificial config block and reboot chain with it, i suppose wal does not have knowledge of this "special" block?
yacovm (Thu, 07 Mar 2019 14:09:33 GMT):
if we have several nodes then how can you send a block without it being in the WAL?
guoger (Thu, 07 Mar 2019 14:10:47 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=u27DkggybJ7LDcurN) @yacovm that's me question
yacovm (Thu, 07 Mar 2019 14:12:15 GMT):
why didn't we do swaps in the first place, does anyone remember? @kostas @guoger @C0rWin
guoger (Thu, 07 Mar 2019 14:12:59 GMT):
swapping the config block file?
yacovm (Thu, 07 Mar 2019 14:13:14 GMT):
no, swapping the references in the metadata
yacovm (Thu, 07 Mar 2019 14:13:26 GMT):
i.e - making the ID point to a different certificate
yacovm (Thu, 07 Mar 2019 14:13:29 GMT):
and preserving the ID
guoger (Thu, 07 Mar 2019 14:13:30 GMT):
i personally don't think it's a problem...
yacovm (Thu, 07 Mar 2019 14:13:41 GMT):
I am asking why we didn't do that in the first place
guoger (Thu, 07 Mar 2019 14:13:51 GMT):
cert rotating didn't occur to me until a later point (when you add that integration test)
yacovm (Thu, 07 Mar 2019 14:16:01 GMT):
I think we should consider supporting the swap
yacovm (Thu, 07 Mar 2019 14:16:36 GMT):
are we endangering v1.4.1 due to implementing it?
guoger (Thu, 07 Mar 2019 14:16:46 GMT):
no, it should be fairly simple
guoger (Thu, 07 Mar 2019 14:18:22 GMT):
rebooting with an artificial config block is a bit more complicated I _think_, we need to insert that block into WAL, otherwise, if another late node started (or we add a new node), when it replicates wal data from leader, it'll miss this block
yacovm (Thu, 07 Mar 2019 14:18:59 GMT):
It will not miss the block
yacovm (Thu, 07 Mar 2019 14:19:05 GMT):
It will onboard
guoger (Thu, 07 Mar 2019 14:19:05 GMT):
it may onboard with deliver
yacovm (Thu, 07 Mar 2019 14:19:55 GMT):
It has to
yacovm (Thu, 07 Mar 2019 14:20:14 GMT):
@jyellick what do you say about we implement the swaps?
guoger (Thu, 07 Mar 2019 14:20:53 GMT):
but the problem is you cannot reliably relay wal to recover ledger
guoger (Thu, 07 Mar 2019 14:21:33 GMT):
(which may not be a valid need, but i think we need to think about edge cases)
guoger (Thu, 07 Mar 2019 14:22:17 GMT):
anyway, just throwing out ideas
jyellick (Thu, 07 Mar 2019 14:37:03 GMT):
> are we endangering v1.4.1 due to implementing it?
@yacovm Though I agree this is a real operational problem, the inability to rotate certs on a single node cluster is not a stop-ship to me. This is something we can address in v1.4.1 or v1.4.2
yacovm (Thu, 07 Mar 2019 14:37:26 GMT):
I am more concerned about the temporary failure that locks your cluster
yacovm (Thu, 07 Mar 2019 14:37:44 GMT):
which you pointed out and we knew it was an issue but said it's somehow tolerable
yacovm (Thu, 07 Mar 2019 14:38:07 GMT):
why not we just say that if we have a config update that doesn't change the number of total consenters, *and* you have a change in a single certificate
yacovm (Thu, 07 Mar 2019 14:38:11 GMT):
then you just re-map the IDs
guoger (Thu, 07 Mar 2019 14:38:29 GMT):
CRs merged on master are cherry-picked to 1.4 and ready for review @yacovm @jyellick @C0rWin
guoger (Thu, 07 Mar 2019 14:39:00 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=JjcDyf2GqNR7Y78SF) @yacovm i think this is swapping? what's the diff?
yacovm (Thu, 07 Mar 2019 14:39:50 GMT):
@guoger there is no diff
yacovm (Thu, 07 Mar 2019 14:39:54 GMT):
i am re-stating the idea
guoger (Thu, 07 Mar 2019 14:40:42 GMT):
i could take on this if we decide to roll with it
yacovm (Thu, 07 Mar 2019 14:41:35 GMT):
I suggest we implement this now while we have time....
yacovm (Thu, 07 Mar 2019 14:41:44 GMT):
@jyellick @C0rWin
yacovm (Thu, 07 Mar 2019 14:41:48 GMT):
wdys?
jyellick (Thu, 07 Mar 2019 14:42:10 GMT):
@guoger If we have time... great, are there any other known bugs/issues we are considering for v1.4?
yacovm (Thu, 07 Mar 2019 14:42:15 GMT):
we can always revert
guoger (Thu, 07 Mar 2019 14:42:28 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=XeFgAny5zTM7rnNEC) @yacovm are you referring to Jason's previous message?: `In a 3 node network, in order to add from 3->4 we now need 3/4 nodes available for consensus.`
yacovm (Thu, 07 Mar 2019 14:42:33 GMT):
yes
jyellick (Thu, 07 Mar 2019 14:45:11 GMT):
Do we have a rough size on the amount of work required for 'swapping' certs?
yacovm (Thu, 07 Mar 2019 14:45:23 GMT):
I think an upper bound is like 20 LOC in prod
yacovm (Thu, 07 Mar 2019 14:45:29 GMT):
and an integration test in test
jyellick (Thu, 07 Mar 2019 14:46:00 GMT):
Sounds reasonable to me. I'd agree, let's do our best to get this into v1.4
jyellick (Thu, 07 Mar 2019 14:46:00 GMT):
Sounds reasonable to me. I'd agree, let's do our best to get this into v1.4.0
guoger (Thu, 07 Mar 2019 14:46:08 GMT):
working on it now
C0rWin (Thu, 07 Mar 2019 15:06:01 GMT):
> removing node at the same time, simply because etcd/raft doesn't support it.
if we going to support the swap or single node certificate rotation there will be no need to reconfigure Raft FSM
yacovm (Thu, 07 Mar 2019 15:21:51 GMT):
wait I have a question - @jyellick @guoger
yacovm (Thu, 07 Mar 2019 15:22:43 GMT):
lets say we have a 3 node cluster with nodes `a`, `b`, `c` and we have 100 channels. We update the TLS certificate in 50 channels to swap `c` with `d` and suddenly, `a` crashes.
yacovm (Thu, 07 Mar 2019 15:22:59 GMT):
now in the first 50 channels we have only `b` and `d` in the config
yacovm (Thu, 07 Mar 2019 15:23:45 GMT):
it means that we have to complete the process with the remaining 50 channels... right?
yacovm (Thu, 07 Mar 2019 15:23:54 GMT):
there is no turning back now for `d`
yacovm (Thu, 07 Mar 2019 15:24:05 GMT):
I guess I need to document this thoroughly.
yacovm (Thu, 07 Mar 2019 15:25:06 GMT):
but - what happens if from some reason we cannot do the config update in the remaining 50 channels?
yacovm (Thu, 07 Mar 2019 15:25:39 GMT):
this can happen, right?
yacovm (Thu, 07 Mar 2019 15:25:54 GMT):
if it happens then we cannot change our minds for the first 50 channels
yacovm (Thu, 07 Mar 2019 15:26:21 GMT):
as we already changed ourselves to be a different certificate
C0rWin (Thu, 07 Mar 2019 15:26:35 GMT):
I think that was the key concern behind doing it gradually and making only one step at a time
C0rWin (Thu, 07 Mar 2019 15:27:51 GMT):
> this can happen, right?
everything can happen, especially something which leads to existantial failures :)
C0rWin (Thu, 07 Mar 2019 15:30:08 GMT):
> now in the first 50 channels we have only `b` and `d` in the config
you have one faulty node, and it's not really a `d` it's `c'` as you only replaced the certificate
C0rWin (Thu, 07 Mar 2019 15:30:48 GMT):
> it means that we have to complete the process with the remaining 50 channels... right?
but that's you were doing at any rate, no? I mean you started it to replace the certificate at first place
C0rWin (Thu, 07 Mar 2019 15:31:34 GMT):
what I'm not sure is how proposed swap operation should look like?
C0rWin (Thu, 07 Mar 2019 15:31:45 GMT):
@guoger @yacovm @jyellick ?
C0rWin (Thu, 07 Mar 2019 15:32:24 GMT):
suppose we going to replaced certificate for node `c`
C0rWin (Thu, 07 Mar 2019 15:32:55 GMT):
the config update proposed where for `c` there is a new certificate
C0rWin (Thu, 07 Mar 2019 15:33:45 GMT):
we can detect it and to the swap, update the metadata, reconfigure comm layer got `a` and `b`
guoger (Thu, 07 Mar 2019 15:34:01 GMT):
this is accurate:
> if we going to support the swap or single node certificate rotation there will be no need to reconfigure Raft FSM
C0rWin (Thu, 07 Mar 2019 15:34:02 GMT):
I guess we need to restart `c` next?
C0rWin (Thu, 07 Mar 2019 15:34:23 GMT):
but back to @yacovm example
C0rWin (Thu, 07 Mar 2019 15:34:33 GMT):
if a goes out of service
C0rWin (Thu, 07 Mar 2019 15:34:33 GMT):
if `a` goes out of service
C0rWin (Thu, 07 Mar 2019 15:34:46 GMT):
the first batch of 50 channel have lost the quorum, no?
C0rWin (Thu, 07 Mar 2019 15:34:48 GMT):
:/
yacovm (Thu, 07 Mar 2019 15:36:12 GMT):
I added this to the administration doc:
yacovm (Thu, 07 Mar 2019 15:36:17 GMT):
Clipboard - March 7, 2019 5:36 PM
C0rWin (Thu, 07 Mar 2019 15:36:41 GMT):
`as fast as possible` :joy:
yacovm (Thu, 07 Mar 2019 15:36:50 GMT):
I can implement multiple TLS certificate support though...
yacovm (Thu, 07 Mar 2019 15:37:02 GMT):
that you also use certificate `c` and `c'`
C0rWin (Thu, 07 Mar 2019 15:37:16 GMT):
> I can implement multiple TLS certificate support though...
one node will have several connections per certificate?
yacovm (Thu, 07 Mar 2019 15:37:23 GMT):
but it is not clear how to integrate it with the same 7050 listener
C0rWin (Thu, 07 Mar 2019 15:37:30 GMT):
yeah
yacovm (Thu, 07 Mar 2019 15:37:40 GMT):
> one node will have several connections per certificate?
no. when you get a TLS client hello, you respond with a TLS server hello with the specified cert
yacovm (Thu, 07 Mar 2019 15:37:49 GMT):
it's supported in recent versions of golang TLS
yacovm (Thu, 07 Mar 2019 15:37:54 GMT):
(god bless, no bugs
)
yacovm (Thu, 07 Mar 2019 15:37:54 GMT):
(god bless, no bugs)
C0rWin (Thu, 07 Mar 2019 15:38:18 GMT):
I think we need to have it
yacovm (Thu, 07 Mar 2019 15:38:29 GMT):
we can only have it for the separate listener
C0rWin (Thu, 07 Mar 2019 15:38:46 GMT):
since otherwise in your scenrario it's not clear how to finish certificate update in presence of single failure
yacovm (Thu, 07 Mar 2019 15:38:59 GMT):
I think we have bigger problems now at the moment, though you obviously agree
C0rWin (Thu, 07 Mar 2019 15:39:01 GMT):
> we can only have it for the separate listener
:(
guoger (Thu, 07 Mar 2019 15:41:41 GMT):
Just to make sure i understand you correctly, the scenario you described is:
- we have node 1, 2, 3 with certs
yacovm (Thu, 07 Mar 2019 15:42:02 GMT):
no no
yacovm (Thu, 07 Mar 2019 15:42:06 GMT):
the problem is the channels
yacovm (Thu, 07 Mar 2019 15:42:27 GMT):
you might get stuck at channel config update because some admin is, say - not cooperating, or jut went to lunch and never came back
yacovm (Thu, 07 Mar 2019 15:42:39 GMT):
you rotated 50 channels
yacovm (Thu, 07 Mar 2019 15:42:46 GMT):
but the remaining 50 - you need that admin
yacovm (Thu, 07 Mar 2019 15:43:17 GMT):
now node 1 crashes. you have to bring node 3 to a consistent state in all channels - either have it have C, or D in all channels
yacovm (Thu, 07 Mar 2019 15:44:23 GMT):
but you can't change it back in the remaining 50 even if the admin of these 50 is there, because you just rotated yourself in these 50 channels.
if you restart, you lose the upper half of the 50 channels
yacovm (Thu, 07 Mar 2019 15:44:23 GMT):
but you can't change it back in the remaining 50 even if the admin of these 50 is there, because you just rotated yourself in these 50 channels.
if you restart as D, you lose the upper half of the 50 channels
yacovm (Thu, 07 Mar 2019 15:44:40 GMT):
because you're still C in them
yacovm (Thu, 07 Mar 2019 15:44:44 GMT):
and 1 is gone now
yacovm (Thu, 07 Mar 2019 15:45:13 GMT):
i guess you can restart as D
yacovm (Thu, 07 Mar 2019 15:45:16 GMT):
migrate back to C
yacovm (Thu, 07 Mar 2019 15:45:20 GMT):
in the first 50 channels
yacovm (Thu, 07 Mar 2019 15:45:29 GMT):
and then wait for the admin to come back from lunch?
yacovm (Thu, 07 Mar 2019 15:48:14 GMT):
added:
yacovm (Thu, 07 Mar 2019 15:48:18 GMT):
Clipboard - March 7, 2019 5:48 PM
jyellick (Thu, 07 Mar 2019 15:49:47 GMT):
So, for prod networks, seems like we should suggest 5+ nodes? That solves most of these problems, no?
jyellick (Thu, 07 Mar 2019 15:50:31 GMT):
If your FT=1, then any maintenance takes your FT to 0. So, it's not exactly a serviceable setup.
yacovm (Thu, 07 Mar 2019 15:50:34 GMT):
yes but it costs $$
jyellick (Thu, 07 Mar 2019 15:50:42 GMT):
Sure.... we should do our best to address it.
yacovm (Thu, 07 Mar 2019 15:51:08 GMT):
I'm not sure
jyellick (Thu, 07 Mar 2019 15:51:09 GMT):
Just saying, we cannot solve the fact that if F=1, and you take a node down for maintenance, bad things can happen in the event of an additional failure.
jyellick (Thu, 07 Mar 2019 15:52:22 GMT):
It's also perhaps not ideal, but we could also suggest that you simply provision a new OSN and sunset the old in order to do cert rotation.
jyellick (Thu, 07 Mar 2019 15:52:28 GMT):
IE, do not do it in place, add and remove.
yacovm (Thu, 07 Mar 2019 15:52:47 GMT):
you can't provision the new OSN if you lost a quorum
jyellick (Thu, 07 Mar 2019 15:53:06 GMT):
Right... I'm suggesting you actually expand the network to n+1 first.
yacovm (Thu, 07 Mar 2019 15:53:12 GMT):
oh
yacovm (Thu, 07 Mar 2019 15:53:20 GMT):
that is terribly expensive
jyellick (Thu, 07 Mar 2019 15:53:20 GMT):
The whole point of these k8s services is supposed to be elasticity.
yacovm (Thu, 07 Mar 2019 15:53:29 GMT):
it means you need to pull gigabytes of data
yacovm (Thu, 07 Mar 2019 15:53:32 GMT):
just to do that
yacovm (Thu, 07 Mar 2019 15:54:13 GMT):
hey look at the bright side.... at least you don't have kafka and zookeeper anymore right?
jyellick (Thu, 07 Mar 2019 15:54:31 GMT):
I guess I don't know what the exact costs are here... maybe I could take this to the platform guys and get their take.
jyellick (Thu, 07 Mar 2019 15:55:06 GMT):
We're not the first CFT service to have to deal with rolling upgrades, I'm sure there are best practices.
yacovm (Thu, 07 Mar 2019 15:55:25 GMT):
putting a blockchain and security always complicates things
jyellick (Thu, 07 Mar 2019 15:55:52 GMT):
Sure... but we're still only CFT, I'd say it's worth checking.
guoger (Thu, 07 Mar 2019 15:57:19 GMT):
@yacovm so.. what i was describing was another actual problem i guess? (which also can be mitigated by having 5 nodes)
yacovm (Thu, 07 Mar 2019 15:58:21 GMT):
@guoger no, it will be solved magically
yacovm (Thu, 07 Mar 2019 15:58:43 GMT):
if the block has been cut
yacovm (Thu, 07 Mar 2019 15:58:52 GMT):
then it means that it was consented on
yacovm (Thu, 07 Mar 2019 15:59:11 GMT):
if node 1 crashes
yacovm (Thu, 07 Mar 2019 15:59:17 GMT):
then it means node 3 cannot crash
yacovm (Thu, 07 Mar 2019 15:59:21 GMT):
right?
yacovm (Thu, 07 Mar 2019 15:59:38 GMT):
so it must have received the block, no?
guoger (Thu, 07 Mar 2019 15:59:38 GMT):
it does not crash, but may not get the config block that rotates cert
yacovm (Thu, 07 Mar 2019 15:59:51 GMT):
you're saying that the leader (node 2) stops speaking to it?
guoger (Thu, 07 Mar 2019 16:00:04 GMT):
this is when 1 was leader
yacovm (Thu, 07 Mar 2019 16:00:26 GMT):
hmmm i see what you mean.
guoger (Thu, 07 Mar 2019 16:00:28 GMT):
and since 2 and 3 cannot talk to each other after `configureComm`, no leader can be elected
yacovm (Thu, 07 Mar 2019 16:00:50 GMT):
yeah, it's another problem
guoger (Thu, 07 Mar 2019 16:02:49 GMT):
but anyway, we know there are problems with rotating certs while node is crashed, but I tend to agree with Jason that
- there probably is best practice to solve this
- having more nodes mitigate risks at higher cost, which is a tradeoff
- we can say: "admins, do your job while reconfiguring it!"
- and we try to solve this after 1.4.1
guoger (Thu, 07 Mar 2019 16:02:49 GMT):
but anyway, we know there are problems with rotating certs while node is crashed, but I tend to agree with Jason that
- there probably is best practice to solve this (or even rolling upgrade 101 :P
- having more nodes mitigate risks at higher cost, which is a tradeoff
- we can say: "admins, do your job while reconfiguring it!"
- and we try to solve this after 1.4.1
yacovm (Thu, 07 Mar 2019 16:03:30 GMT):
I say we document this and just say they should really run 5 nodes to be bulletproof
adarshsaraf123 (Thu, 07 Mar 2019 16:14:51 GMT):
Screenshot 2019-03-07 at 9.43.58 PM.png
adarshsaraf123 (Thu, 07 Mar 2019 16:15:12 GMT):
Is this a known intergration test flake?
yacovm (Thu, 07 Mar 2019 16:22:29 GMT):
@guoger is it possible to Yield() the leadership in Raft ?
yacovm (Thu, 07 Mar 2019 16:22:35 GMT):
i mean, in the Raft node API
yacovm (Thu, 07 Mar 2019 16:22:42 GMT):
I recall that yes?
guoger (Thu, 07 Mar 2019 16:22:48 GMT):
Transfer leadership? Yes
yacovm (Thu, 07 Mar 2019 16:23:10 GMT):
`TransferLeadership`
yacovm (Thu, 07 Mar 2019 16:23:11 GMT):
yeah
yacovm (Thu, 07 Mar 2019 16:23:28 GMT):
when we rotate the TLS cert of the leader it will make the nodes lose leadership
yacovm (Thu, 07 Mar 2019 16:23:33 GMT):
I guess i can document it too
yacovm (Thu, 07 Mar 2019 16:24:17 GMT):
but wondering if it is possible to solve it somehow. say the node that is the leader understands its cert is rotated
yacovm (Thu, 07 Mar 2019 16:24:28 GMT):
so it can transfer leadership and reject the change
yacovm (Thu, 07 Mar 2019 16:24:43 GMT):
this way you never rotate a leader's TLS cert
guoger (Thu, 07 Mar 2019 16:36:37 GMT):
right, and others should wait for leader change before reconfiguring comm, if they see that leader's cert is being rotated
guoger (Thu, 07 Mar 2019 16:36:37 GMT):
right, and ~others~ all should wait for leader change before reconfiguring comm, if they see that leader's cert is being rotated
guoger (Thu, 07 Mar 2019 16:37:13 GMT):
this is doable, but let's do incremental changes.
guoger (Thu, 07 Mar 2019 16:37:25 GMT):
about to push the CR. have you created a JIRA for this already? @yacovm
guoger (Thu, 07 Mar 2019 16:41:36 GMT):
created one here https://jira.hyperledger.org/browse/FAB-14539, let me know if it's duplicate
guoger (Thu, 07 Mar 2019 16:47:21 GMT):
and created https://jira.hyperledger.org/browse/FAB-14540 to track leader transfer
guoger (Thu, 07 Mar 2019 16:55:30 GMT):
I’ll do that tomorrow though... should also be simple enough since we have the mechanism to block request and wait for a change
adarshsaraf123 (Thu, 07 Mar 2019 17:15:28 GMT):
@C0rWin @guoger @jyellick @yacovm CR to add additional metrics for etcdraft: https://gerrit.hyperledger.org/r/#/c/29926/
Would be good if we can have this merged soon so that it can also be cherry-picked to v1.4.1
adarshsaraf123 (Thu, 07 Mar 2019 17:15:28 GMT):
@C0rWin @guoger @jyellick @yacovm CR to add additional metrics for etcdraft: https://gerrit.hyperledger.org/r/#/c/29926/
Would be good if we can have this reviewed soon so that it can also be cherry-picked to v1.4.1
mauricio (Thu, 07 Mar 2019 21:04:30 GMT):
Has joined the channel.
guoger (Fri, 08 Mar 2019 06:13:38 GMT):
@yacovm is it possible to get _the node with longest connected time_ from communication? it is useful to decide whom to transfer leadership to. It's not urgent though, since this can be indirectly got from raft FSM using `RecentActive bool`. However, it's more accurate and appropriate to acquire such knowledge from communication layer IMO. If you think it's doable and adds value, i could create a JIRA in our backlog.
yacovm (Fri, 08 Mar 2019 07:18:22 GMT):
No it is not
yacovm (Fri, 08 Mar 2019 07:20:12 GMT):
RecentActive sounds better to me
guoger (Fri, 08 Mar 2019 10:11:10 GMT):
@adarshsaraf123 if unit is second, it would still show float number, right? like 0.02sec, but not rounded to 0
adarshsaraf123 (Fri, 08 Mar 2019 10:41:36 GMT):
true.
adarshsaraf123 (Fri, 08 Mar 2019 10:41:36 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=LxZ9uS54Xxnt96ZpG) @guoger Yes that's right. I am making the change to seconds/
adarshsaraf123 (Fri, 08 Mar 2019 10:41:36 GMT):
Yes that's right. I am making the change to seconds.
tock (Fri, 08 Mar 2019 12:43:27 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=wxxjpprzQ7ZsPWuRY) @yacovm or add a node before maintenance... it must be cheaper to add a node for a couple of days every few months then run 5 all year...
yacovm (Fri, 08 Mar 2019 13:54:30 GMT):
> it must be cheaper to add a node for a couple of days every few months then run 5 all year...
I know ;)
guoger (Fri, 08 Mar 2019 17:28:42 GMT):
i've seen this UT flake more frequently on 1.4 branch, is there a CR on master that should be cherry-picked for this? https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/10227/console
yacovm (Fri, 08 Mar 2019 17:32:20 GMT):
no
jyellick (Mon, 11 Mar 2019 01:04:55 GMT):
A reminder to all for tomorrow morning's scrum, the US transitioned to daylight saving time over the weekend, so please confirm that your calendars match up with 8:15 EDT for the scrum.
jyellick (Mon, 11 Mar 2019 04:03:54 GMT):
It looks to me like we are persisting the RaftMetadata in the block metadata with each block, am I reading this incorrectly? Won't this mean that we're storing a copy of all of the TLS certs in every block? Who would be the right person to talk to about our crash/stop recovery path?
guoger (Mon, 11 Mar 2019 04:15:36 GMT):
@jyellick yes we are persisting TLS certs in every block as part of block metadata
guoger (Mon, 11 Mar 2019 04:16:46 GMT):
we can chat here so others can chime in. I'm also open to a call though, if you prefer
jyellick (Mon, 11 Mar 2019 04:28:18 GMT):
@guoger Chat works for me. What is the rationale for putting the TLS certs into every block. They can only change at the config block, no?
guoger (Mon, 11 Mar 2019 04:31:50 GMT):
yes they are *only* changed at config block. However, when a chainsupport is created, meta data is grabbed from *last block*, instead of *last config block*, [here](https://github.com/hyperledger/fabric/blob/70957926de5dbf1cae57747ae54161c693d95caa/orderer/common/multichannel/chainsupport.go#L45)
jyellick (Mon, 11 Mar 2019 04:32:24 GMT):
Correct, but there is also the `OrdererConfig` which gives the `ConsensusMetadata`
jyellick (Mon, 11 Mar 2019 04:32:45 GMT):
(Which is from the last config block)
jyellick (Mon, 11 Mar 2019 04:36:20 GMT):
In fact the code gets this value already, but prefers the metadata encoded in the block for some reason:
https://github.com/hyperledger/fabric/blob/3c8160648875665d1a13e1309c201de6b2e5ead5/orderer/consensus/etcdraft/consenter.go#L133-L136
https://github.com/hyperledger/fabric/blob/3c8160648875665d1a13e1309c201de6b2e5ead5/orderer/consensus/etcdraft/consenter.go#L154
https://github.com/hyperledger/fabric/blob/3c8160648875665d1a13e1309c201de6b2e5ead5/orderer/consensus/etcdraft/consenter.go#L224-L243
jyellick (Mon, 11 Mar 2019 04:36:20 GMT):
In fact the code gets this value already, but prefers the metadata encoded in the latest block for some reason:
https://github.com/hyperledger/fabric/blob/3c8160648875665d1a13e1309c201de6b2e5ead5/orderer/consensus/etcdraft/consenter.go#L133-L136
https://github.com/hyperledger/fabric/blob/3c8160648875665d1a13e1309c201de6b2e5ead5/orderer/consensus/etcdraft/consenter.go#L154
https://github.com/hyperledger/fabric/blob/3c8160648875665d1a13e1309c201de6b2e5ead5/orderer/consensus/etcdraft/consenter.go#L224-L243
jyellick (Mon, 11 Mar 2019 04:37:42 GMT):
I would expect that it's impossible for these two values to differ, am I missing something?
guoger (Mon, 11 Mar 2019 04:39:25 GMT):
:thinking: that's looks correct to me. I think it's safe not to persist certs in normal blocks... @C0rWin am I missing anyting?
guoger (Mon, 11 Mar 2019 04:39:25 GMT):
:thinking: that's looks correct to me. I think it's safe not to persist certs in normal blocks... @C0rWin am I missing anything?
jyellick (Mon, 11 Mar 2019 04:41:15 GMT):
I would have expected perhaps the index and or term to be persisted in the block metadata. I realize we have the WAL though, so perhaps that is unnecessary.
guoger (Mon, 11 Mar 2019 04:41:56 GMT):
we are persisting Raft index to avoid replay of blocks
jyellick (Mon, 11 Mar 2019 04:42:43 GMT):
Ah, that's right, I did see that being updated just before writing the block.
guoger (Mon, 11 Mar 2019 04:44:11 GMT):
I think persisting certs is not necessary. Is it performance you were concerned about? we could get @C0rWin to confirm and remove it
jyellick (Mon, 11 Mar 2019 04:44:45 GMT):
Partially performance, but actually the long term storage penalty worries me most.
guoger (Mon, 11 Mar 2019 05:14:08 GMT):
@jyellick `ConsensusMetadata` actually returns [user-defined metadata](https://github.com/hyperledger/fabric/blob/3c8160648875665d1a13e1309c201de6b2e5ead5/protos/orderer/etcdraft/configuration.proto#L14-L19), but we are persisting [RaftMetadata](https://github.com/hyperledger/fabric/blob/3c8160648875665d1a13e1309c201de6b2e5ead5/protos/orderer/etcdraft/configuration.proto#L41-L55) that helps us track the status of chain
guoger (Mon, 11 Mar 2019 05:15:37 GMT):
we can still skip persisting certs to normal blocks. but we need to read RaftMetadata, instead of config metadata, from last config block. (and of course get the raft index from latest block)
guoger (Mon, 11 Mar 2019 05:15:37 GMT):
we can still skip persisting certs to normal blocks. but we need to read RaftMetadata, instead of config metadata, from *last config block*. (and of course get the raft index from *last block*)
guoger (Mon, 11 Mar 2019 05:16:41 GMT):
the main difference there is *config metadata* stores a list of certs w/o raft ID mapping, but *RaftMetadata* does
C0rWin (Mon, 11 Mar 2019 08:28:49 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=vKa25ifKMbLcP9kLn) @jyellick that was call made by @kostas at design time, obviously we should have store this only per config block only
yacovm (Mon, 11 Mar 2019 08:29:53 GMT):
btw why is
```
# Max Message Count: The maximum number of messages to permit in a
# batch. No block will contain more than this number of messages.
MaxMessageCount: 10
```
set to 10? I think it should be a high value by default, like 500.
C0rWin (Mon, 11 Mar 2019 08:30:54 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=nvaPBtmTZKeQcKnkj) @guoger we should be safe, but I'm sure there was some rational thinking behind the design, not sure if we can ask @kostas at this point in time, but I remember he gave me arguments why he decided this
yacovm (Mon, 11 Mar 2019 08:31:33 GMT):
I think the reason was that we don't store the id mapping in the config, but we implicitly calculate them
yacovm (Mon, 11 Mar 2019 08:31:39 GMT):
but we should've just stored hashes
yacovm (Mon, 11 Mar 2019 08:31:53 GMT):
if you have the last config block, you have the latest certs.
yacovm (Mon, 11 Mar 2019 08:32:12 GMT):
and if you just store hashes of the certs in the mapping, it reduces the size from 700 bytes to 32 bytes
C0rWin (Mon, 11 Mar 2019 08:32:22 GMT):
no-no, I remember the was something else behind it, let try to digg in the chat history
C0rWin (Mon, 11 Mar 2019 08:32:22 GMT):
no-no, I remember it was something else behind it, let try to digg in the chat history
C0rWin (Mon, 11 Mar 2019 08:32:22 GMT):
no-no, I remember it was something else behind it, let me try to digg in the chat history
C0rWin (Mon, 11 Mar 2019 08:33:46 GMT):
and btw, I think we need to refactor and re-name `ConsensusMetadata` and `RaftMetadata` as it raise a lot of confusion
tock (Mon, 11 Mar 2019 10:59:11 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=CqBnZZ7WQuWfC6DZP) @yacovm I agree. A quick fix to the storage issue would be to store the SHA256 of the certificate together with the node ID.
tock (Mon, 11 Mar 2019 11:00:34 GMT):
@yacovm are you familiar with this UT failure in cluster? https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/10280/console
tock (Mon, 11 Mar 2019 11:02:14 GMT):
and another one: https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/10279/console
guoger (Mon, 11 Mar 2019 11:04:14 GMT):
I don’t get the idea of storing hash though. Why is it needed?
yacovm (Mon, 11 Mar 2019 11:05:13 GMT):
yes
yacovm (Mon, 11 Mar 2019 11:05:17 GMT):
@tock
yacovm (Mon, 11 Mar 2019 11:05:28 GMT):
@guoger the hash is to store only 32 bytes instead of 700, per consenter node
guoger (Mon, 11 Mar 2019 11:06:50 GMT):
I mean why do we need to store something at all? If we can get consented mapping and certs from last config block
guoger (Mon, 11 Mar 2019 11:07:02 GMT):
Consenter mapping
yacovm (Mon, 11 Mar 2019 11:07:10 GMT):
you don't get the IDs from the last block
yacovm (Mon, 11 Mar 2019 11:07:21 GMT):
to compute the IDs you need all config blocks since ever
guoger (Mon, 11 Mar 2019 11:08:04 GMT):
Why not, it’s persisted in every block currently, no?
yacovm (Mon, 11 Mar 2019 11:09:01 GMT):
we are talking past each other
yacovm (Mon, 11 Mar 2019 11:09:36 GMT):
you are saying we can get consenter mapping and certs from last config block
yacovm (Mon, 11 Mar 2019 11:09:39 GMT):
but it is not what we do
yacovm (Mon, 11 Mar 2019 11:09:44 GMT):
we don't store the IDs in the config
guoger (Mon, 11 Mar 2019 11:17:27 GMT):
Really? Isn’t it part of ordered metadata, where we have a map of ID to certs?
guoger (Mon, 11 Mar 2019 11:17:40 GMT):
In config, we only have a slice
guoger (Mon, 11 Mar 2019 11:27:34 GMT):
basically [this proto](https://github.com/hyperledger/fabric/blob/3c8160648875665d1a13e1309c201de6b2e5ead5/protos/orderer/etcdraft/configuration.proto#L44) is persisted in every block currently, and it does not change in normal blocks. so we should be able to get this just from last config block. unless @C0rWin can trace back to the reason behind this decision. what am i missing?
yacovm (Mon, 11 Mar 2019 11:39:45 GMT):
that's what I am saying... we are storing in the config block, the slices and not the mapping
C0rWin (Mon, 11 Mar 2019 11:44:18 GMT):
@guoger so basically the reason behind this decision is within `chainsupport.go` and the way we call `HandleChain` API
C0rWin (Mon, 11 Mar 2019 11:44:40 GMT):
chainsupport is not consensus type aware
C0rWin (Mon, 11 Mar 2019 11:45:10 GMT):
hence extracts metadata information to pass it to the `HandleChain` both for Kafka and Raft
C0rWin (Mon, 11 Mar 2019 11:45:28 GMT):
in kafka it makes sense since in each block you keep offsets
C0rWin (Mon, 11 Mar 2019 11:46:09 GMT):
while for Raft this could be optimized, but that mean we need to refactor chainsupport and extract metadata based on consensus type
C0rWin (Mon, 11 Mar 2019 11:46:12 GMT):
or even better
C0rWin (Mon, 11 Mar 2019 11:46:38 GMT):
each consenter should be able to retreive metadata itself w/o chainsupport doing job for them
C0rWin (Mon, 11 Mar 2019 11:47:25 GMT):
since this line required some substential amount of refactoring, it was decided to continue with approach as proposed by @kostas in his design doc
C0rWin (Mon, 11 Mar 2019 11:47:44 GMT):
we can do this optimization, though I think it's a bit risky at this point in time
C0rWin (Mon, 11 Mar 2019 11:48:14 GMT):
probably won't be able to make this improvement for v1.4.1 while we can do it for v2.0 and make a drop for v1.4.2 later
C0rWin (Mon, 11 Mar 2019 11:49:09 GMT):
basically we can discuss it during the scrum today and decide whenever to contain and make this fix now
C0rWin (Mon, 11 Mar 2019 11:49:18 GMT):
@jyellick @yacovm ^^^^
C0rWin (Mon, 11 Mar 2019 11:52:32 GMT):
> that's what I am saying... we are storing in the config block, the slices and not the mapping
@yacovm, while this is true, we do store in block's metadata the mapping as well (same as we do for all blocks we are processing)
yacovm (Mon, 11 Mar 2019 11:59:03 GMT):
yes, I am not saying we are not
tock (Mon, 11 Mar 2019 13:20:36 GMT):
@guoger I have question on:
```
// Validate the config update for being of Type A or Type B as described in the design doc.
func (c *Chain) checkConfigUpdateValidity(ctx *common.Envelope) error {
... // Check that only the ConsensusType is updated in the write-set
if ordererConfigGroup, ok := configUpdate.WriteSet.Groups["Orderer"]; ok {
if val, ok := ordererConfigGroup.Values["ConsensusType"]; ok {
return c.checkConsentersSet(val)
}
}
return nil
default:
return errors.Errorf("config transaction has unknown header type")
}
}
```
tock (Mon, 11 Mar 2019 13:20:36 GMT):
@guoger I have question on:
```
// Validate the config update for being of Type A or Type B as described in the design doc.
func (c *Chain) checkConfigUpdateValidity(ctx *common.Envelope) error {
... // Check that only the ConsensusType is updated in the write-set
if ordererConfigGroup, ok := configUpdate.WriteSet.Groups["Orderer"]; ok {
if val, ok := ordererConfigGroup.Values["ConsensusType"]; ok {
return c.checkConsentersSet(val)
}
}
return nil
...
}
}
```
The check in there is not really doing what the comment is saying, right?
tock (Mon, 11 Mar 2019 13:20:36 GMT):
@guoger I have a question on:
```
// Validate the config update for being of Type A or Type B as described in the design doc.
func (c *Chain) checkConfigUpdateValidity(ctx *common.Envelope) error {
... // Check that only the ConsensusType is updated in the write-set
if ordererConfigGroup, ok := configUpdate.WriteSet.Groups["Orderer"]; ok {
if val, ok := ordererConfigGroup.Values["ConsensusType"]; ok {
return c.checkConsentersSet(val)
}
}
return nil
...
}
}
```
The check in there is not really doing what the comment is saying, right?
tock (Mon, 11 Mar 2019 13:22:57 GMT):
It is allowed to do additional config changes as well, correct? It just checking the `"ConsensusType"` portion of the config, right?
tock (Mon, 11 Mar 2019 13:22:57 GMT):
It is allowed to do additional config changes as well, correct? It is just checking the `"ConsensusType"` portion of the config, right?
guoger (Mon, 11 Mar 2019 13:25:25 GMT):
right. ultimately this just checks the validity of type b config
tock (Mon, 11 Mar 2019 13:26:23 GMT):
so a better comment would probably say `Check only the ConsensusType update in the write-set`
tock (Mon, 11 Mar 2019 13:27:26 GMT):
(In addition Type A and Type B is a bit cryptic, I would change that to a clearer description...)
tock (Mon, 11 Mar 2019 13:28:39 GMT):
I am looking at this part of the code because I am working on https://jira.hyperledger.org/browse/FAB-14052
jyellick (Mon, 11 Mar 2019 13:58:51 GMT):
I've been looking more at this consensus metadata stuff. Why don't we simply modify `etcdraft.RaftMetadata` to instead of contain a `map[raftID]TLSCert` to be a `[]raftIDs`. We have a slice of consenters in the channel config which has an order which is preserved and modified only at config updates. We also have a 1-1 correspondance between consenter and raftID, which also only changes on config update. Seems simple to persist the raft ID for each consenter, by slice position. It would be a very small code change and reduce the size of the mapping from n*(varint IDs + TLSCerts) to n*(var-int encoded IDs.)
yacovm (Mon, 11 Mar 2019 14:00:13 GMT):
you mean `[]uint64` right?
jyellick (Mon, 11 Mar 2019 14:00:21 GMT):
Right
yacovm (Mon, 11 Mar 2019 14:00:26 GMT):
it's possible
yacovm (Mon, 11 Mar 2019 14:01:31 GMT):
we can also use gamma coding
yacovm (Mon, 11 Mar 2019 14:01:34 GMT):
it's more efficient
yacovm (Mon, 11 Mar 2019 14:01:38 GMT):
(just kidding)
jyellick (Mon, 11 Mar 2019 14:02:47 GMT):
But I think this would be a pretty minimal change, and even more space efficient than hashes. Probably sufficiently so that it wouldn't be worth changing the ChainSupport interface and we could contain this for v1.4.1. In fact, I'll volunteer to do the work if we can agree on this approach.
yacovm (Mon, 11 Mar 2019 14:03:03 GMT):
you mean that the index `i` in the metadata denotes the ID of the consenter in the `i`'th index in the consenter slice in the config, right?
jyellick (Mon, 11 Mar 2019 14:03:08 GMT):
Yes
yacovm (Mon, 11 Mar 2019 14:05:52 GMT):
I'm for it
C0rWin (Mon, 11 Mar 2019 14:06:00 GMT):
sounds reasonable
guoger (Mon, 11 Mar 2019 14:21:47 GMT):
do we still need other fields in this case?
```
// Carries the Raft ID value that will be assigned
// to the next OSN that will join this cluster.
uint64 next_consenter_id = 2;
// Raft cluster configurations count
uint64 conf_change_counts = 3;
```
guoger (Mon, 11 Mar 2019 14:22:42 GMT):
i guess yes, right? (just to confirm here)
tock (Mon, 11 Mar 2019 14:23:15 GMT):
I think you definitely need the first
tock (Mon, 11 Mar 2019 14:23:15 GMT):
I think you definitely need the `next_consenter_id`
guoger (Mon, 11 Mar 2019 14:26:08 GMT):
actually.. what's the purpose of `conf_change_counts`? i see it's being used as raft id? @C0rWin
C0rWin (Mon, 11 Mar 2019 14:27:17 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=FJo972dAzbua2NiDo) @guoger it used for Raft config proposal
adarshsaraf123 (Mon, 11 Mar 2019 14:30:17 GMT):
To elaborate on `conf_change_counts` (and @C0rWin please correct me if I am wrong), i believe etcdraft requires every conf change proposal to have an id which we assign as `conf_change_counts`.
adarshsaraf123 (Mon, 11 Mar 2019 14:30:17 GMT):
To elaborate on `conf_change_counts` (and @C0rWin please correct me if I am wrong), i believe etcdraft requires every conf change proposal to have an id which we assign as `conf_change_counts` (and it is not being used as raft id).
guoger (Mon, 11 Mar 2019 14:35:31 GMT):
got it. we actually don't need it...
guoger (Mon, 11 Mar 2019 14:35:59 GMT):
it's basically a unique ID generated so that etcd can keep track of a request
guoger (Mon, 11 Mar 2019 14:36:25 GMT):
etcd does this:
```
// Register waits returns a chan that waits on the given ID.
// The chan will be triggered when Trigger is called with
// the same ID.
```
guoger (Mon, 11 Mar 2019 14:37:48 GMT):
we block incoming requests till this confchange is applied with our own mechanism (and we also make sure it is applied upon failover)
guoger (Mon, 11 Mar 2019 14:38:19 GMT):
etcd/raft doesn't use this field
C0rWin (Mon, 11 Mar 2019 14:38:22 GMT):
so are you saying we can get rid of this?
guoger (Mon, 11 Mar 2019 14:38:28 GMT):
yep
C0rWin (Mon, 11 Mar 2019 14:41:21 GMT):
@guoger do you mind to open a JIRA and I will take care of it?
guoger (Mon, 11 Mar 2019 14:45:48 GMT):
sure
guoger (Mon, 11 Mar 2019 14:46:08 GMT):
i delete references of this and UT passed at least
C0rWin (Mon, 11 Mar 2019 14:46:45 GMT):
just completed the same, but making changes to the protos
yacovm (Mon, 11 Mar 2019 16:00:24 GMT):
https://gerrit.hyperledger.org/r/#/c/29996/
https://gerrit.hyperledger.org/r/#/c/29997/
@jyellick @C0rWin ^
jyellick (Mon, 11 Mar 2019 16:12:01 GMT):
``` MaxInflightMsgs: 5```
jyellick (Mon, 11 Mar 2019 16:12:26 GMT):
@yacovm @guoger @C0rWin I thought we had discussed '10' here? I'm fine with 5... just want to make sure this was intentional.
yacovm (Mon, 11 Mar 2019 16:13:30 GMT):
ah, so - the default egress queue size in the communication is 10 messages
yacovm (Mon, 11 Mar 2019 16:13:35 GMT):
so i made it 5 to make it lower
yacovm (Mon, 11 Mar 2019 16:13:50 GMT):
if it is 10 and we have 10 outgoing messages circulating
yacovm (Mon, 11 Mar 2019 16:13:58 GMT):
they may all end up in our buffers right?
jyellick (Mon, 11 Mar 2019 16:14:27 GMT):
Ah, makes sense
jyellick (Mon, 11 Mar 2019 18:26:04 GMT):
@guoger (or anyone) I'm trying understand https://github.com/hyperledger/fabric/blob/3c8160648875665d1a13e1309c201de6b2e5ead5/orderer/consensus/etcdraft/chain.go#L966-L975 a little better
jyellick (Mon, 11 Mar 2019 18:26:29 GMT):
We have a config block, we try to get the raft metadata out of it... but then sometimes the raft metadata can be nil?
jyellick (Mon, 11 Mar 2019 18:26:29 GMT):
We have a config block, we try to get the metadata out of it... but then sometimes the metadata can be nil? But this is non-fatal and we keep going
jyellick (Mon, 11 Mar 2019 18:27:38 GMT):
If so... then it looks like `changes` is nil, and then we dereference it
jyellick (Mon, 11 Mar 2019 18:27:38 GMT):
If so... then it looks like `changes` is nil, and then we dereference it -- which I see is actually okay, we check and if so, do nothing
adarshsaraf123 (Mon, 11 Mar 2019 19:05:01 GMT):
@jyellick
> We have a config block, we try to get the metadata out of it... but then sometimes the metadata can be nil?
The `metadata` can be `nil` since a config change may not update the consensus metadata.
Even if the `metadata` is not `nil`, `changes` could still be `nil` since we may have updated the various `options` and not the `consenters`.
This block of code ensures that a node that is catching up reconfigures the communication layer if there have been any changes to the consenter set among the blocks it is catching up with.
adarshsaraf123 (Mon, 11 Mar 2019 19:05:01 GMT):
@jyellick
> We have a config block, we try to get the metadata out of it... but then sometimes the metadata can be nil?
The `metadata` can be `nil` since a config change may not update the consensus metadata.
Even if the `metadata` is not `nil`, `changes` could still be `nil` since we may have updated the various `options` and not the `consenters`.
This block of code ensures that a node that is catching up figures out the need to reconfigure the communication layer owing to changes to the consenter set among the blocks it is catching up with.
adarshsaraf123 (Mon, 11 Mar 2019 19:06:50 GMT):
Hope this helps.
jyellick (Mon, 11 Mar 2019 19:10:45 GMT):
I think it is the structure that is confusing to me. I understand that not all config blocks affect the consensus details.
jyellick (Mon, 11 Mar 2019 19:11:03 GMT):
However, I would expect that we would detect this early, and exit.
jyellick (Mon, 11 Mar 2019 19:11:20 GMT):
I'm trying to understand why we would proceed through these other checks if for instance the metadata is nil.
adarshsaraf123 (Mon, 11 Mar 2019 19:15:06 GMT):
Ah okay. Agree with you :+1_tone4:
jyellick (Mon, 11 Mar 2019 20:06:42 GMT):
```# MaxInflightMsgs limits the max number of in-flight append messages during optimistic replication phase.```
Isn't this a bit disingenuous? Naively, I assumed heartbeat messages would count against this limit, as per spec, the are a type of 'append'. It doesn't make sense to 'optimistically replicate' heartbeats though, so I understand why they would not count. Still, if this number corresponds directly to the number of in flight blocks, wouldn't it be better to simply call it such?
yacovm (Mon, 11 Mar 2019 21:04:31 GMT):
@jyellick this doesn't affect heartbeats
yacovm (Mon, 11 Mar 2019 21:04:33 GMT):
only blocks
yacovm (Mon, 11 Mar 2019 21:05:08 GMT):
> if this number corresponds directly to the number of in flight blocks, wouldn't it be better to simply call it such?
I really have no idea why it was called this way
yacovm (Mon, 11 Mar 2019 22:19:49 GMT):
nor I knew what *disingenuous* meant
guoger (Mon, 11 Mar 2019 23:24:31 GMT):
This is simply how etcdraft pkg calls it. We could certainly expose it with a different name. Raft Config messages are counted in this as well but that’s only one message at a time.
guoger (Mon, 11 Mar 2019 23:26:26 GMT):
As well as other message types used internally by raft. So it only affects inflight blocks in most cases, but sometimes others as well
guoger (Mon, 11 Mar 2019 23:27:19 GMT):
I do agree with renaming user facing option though
guoger (Mon, 11 Mar 2019 23:33:17 GMT):
Do you want me to make the change?
jyellick (Tue, 12 Mar 2019 02:17:14 GMT):
I'd think so, but open to other opinions
KyunghoKim (Tue, 12 Mar 2019 03:10:06 GMT):
Has joined the channel.
yacovm (Tue, 12 Mar 2019 08:30:13 GMT):
yeah i say we can change "messages" to "blocks"
guoger (Tue, 12 Mar 2019 08:42:54 GMT):
https://jira.hyperledger.org/browse/FAB-14593
guoger (Wed, 13 Mar 2019 02:48:49 GMT):
@yacovm what's the diff between _inactive chain_ and _disabled chain_?
jyellick (Wed, 13 Mar 2019 05:25:14 GMT):
@guoger Especially, but others too, could you please take a look at https://gerrit.hyperledger.org/r/c/30051/
jyellick (Wed, 13 Mar 2019 05:26:56 GMT):
It is the end CR in a series of 3 which:
1. Renames the metadata protos to be a bit more intuitive as to their use
2. Refactors the detectConfChange code a bit to make it easier to modify the contents of the protos
3. Changes the slice mapping nodeID->consenter with a simple slide of nodeID, and then builds the map from a combination of the config metadata slice of consenters and the block metadata slice.
jyellick (Wed, 13 Mar 2019 05:29:20 GMT):
The changes ended up being a little more invasive than I was hoping for, but I think/hope they are still fairly straightforward. In addition to any comments on correctness, I'd like opinions from everyone on how good we feel about this going into test now, and whether we feel that it invalidates any of our testing (especially our failure and recovery paths). Obviously it's much easier to change these message before we have them on chains we will have to support indefinitely, but I also don't want us to rush this through simply because we are afraid of future work. So, please, review and consider whether we would like to integrate.
jyellick (Wed, 13 Mar 2019 05:29:48 GMT):
@yacovm @C0rWin @tock @adarshsaraf123 ^
yacovm (Wed, 13 Mar 2019 08:06:18 GMT):
@guoger same thing
yacovm (Wed, 13 Mar 2019 10:32:17 GMT):
@jyellick I am pretty confident in our integration tests, and we can always just ask SVT to retest everything, as they are doing anyway for v1.4.1
yacovm (Wed, 13 Mar 2019 10:33:22 GMT):
we need to look at the disk space saved a year from now which is substantial.
tock (Wed, 13 Mar 2019 16:02:22 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=vyYwgDkMhwtPknkZF) 100 Tx/sec * 5 nodes * 700B * 3600*24*365 ~ 11TB
tock (Wed, 13 Mar 2019 16:02:22 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=vyYwgDkMhwtPknkZF) 100 Tx/sec x 5 nodes x 700B x 3600 x 24 x 365 ~ 11TB
guoger (Wed, 13 Mar 2019 17:00:51 GMT):
@jyellick overall, i think your approach looks good! (and the refactoring streamlines the code quite a bit!) Although there's a problem w.r.t the integration test failure: https://jenkins.hyperledger.org/job/fabric-verify-integration-tests-x86_64/7084/console
I _think_ the problem is, when a node detects its own eviction, it pulls blocks up to a config block, writes them to ledger and exits. It [writes block with](https://github.com/hyperledger/fabric/blob/dbb4451c44095af27c27fbe34aad1d028a982d67/orderer/consensus/etcdraft/util.go#L586) `support.WriteBlock` for both normal blocks and config blocks, which is good because we don't want to actually update configs. However, config sequence is *not* updated, and the `LastConfig` in those blocks do *not* refer to correct config block, instead, an older config block was referred.
guoger (Wed, 13 Mar 2019 17:00:51 GMT):
@jyellick overall, i think your approach looks good! (and the refactoring streamlines the code quite a bit!) Although there's a problem w.r.t the integration test failure: https://jenkins.hyperledger.org/job/fabric-verify-integration-tests-x86_64/7084/console
I _think_ the problem is, when a node detects its own eviction, it pulls blocks up to a config block, writes them to ledger and exits. It [writes block with](https://github.com/hyperledger/fabric/blob/dbb4451c44095af27c27fbe34aad1d028a982d67/orderer/consensus/etcdraft/util.go#L586) `support.WriteBlock` for both normal blocks and config blocks, which ~is good because we don't want to actually update configs~ I'm not sure if it's good. However, config sequence is *not* updated, and the `LastConfig` in those blocks do *not* refer to correct config block, instead, an older config block was referred.
jyellick (Wed, 13 Mar 2019 17:03:37 GMT):
Ah, I see, so it should only be the last config block that's written with `WriteBlock`?
jyellick (Wed, 13 Mar 2019 17:03:52 GMT):
(The others would be written normally because we do want those configs to be updated in that case)
guoger (Wed, 13 Mar 2019 17:03:57 GMT):
In the case of that integration test, when this node is restarted, it loads
- `consenters []*etcdraft.Consenter` from older config block: {o1, o2, o3}
- `ConsenterIds []uint64` from last block, [2, 3]
and crashes
jyellick (Wed, 13 Mar 2019 17:04:32 GMT):
Ah, so actually we had this bug before (wrong last config block in the metadata)
guoger (Wed, 13 Mar 2019 17:04:38 GMT):
right
jyellick (Wed, 13 Mar 2019 17:05:28 GMT):
And what would be wrong with updating the config via `WriteConfigBlock` and updating the configs?
jyellick (Wed, 13 Mar 2019 17:05:28 GMT):
And what would be wrong with writing the block via `WriteConfigBlock` and updating the configs?
guoger (Wed, 13 Mar 2019 17:05:56 GMT):
just updated my post while you ask this :joy:
guoger (Wed, 13 Mar 2019 17:07:30 GMT):
@yacovm any reason why we weren't actually updating configs while pulling blocks during eviction?
guoger (Wed, 13 Mar 2019 17:07:30 GMT):
(if this node is evicted from systemchannel, i think those channel creation config should be no-op?)
guoger (Wed, 13 Mar 2019 17:10:26 GMT):
Other than this, I'd +1 the CR and I think we should integrate :)
yacovm (Wed, 13 Mar 2019 17:54:13 GMT):
why do you need to use the chain method?
yacovm (Wed, 13 Mar 2019 17:54:27 GMT):
the chain is dead once you halt. it is never used again
yacovm (Wed, 13 Mar 2019 17:54:40 GMT):
all you care is pulling the blocks and committing them.
yacovm (Wed, 13 Mar 2019 17:56:25 GMT):
oh
yacovm (Wed, 13 Mar 2019 17:56:51 GMT):
i guess we need to use `chain.writeBlock` instead of `support.WriteBlock`
yacovm (Wed, 13 Mar 2019 17:57:40 GMT):
wait actually
yacovm (Wed, 13 Mar 2019 17:57:42 GMT):
i don't understand
yacovm (Wed, 13 Mar 2019 17:57:48 GMT):
you pull blocks from a consenter
yacovm (Wed, 13 Mar 2019 17:57:55 GMT):
so the metadata and last config block should be correct
yacovm (Wed, 13 Mar 2019 17:58:03 GMT):
because that consenter committed that block correctly
yacovm (Wed, 13 Mar 2019 18:00:31 GMT):
and the chain is halted
yacovm (Wed, 13 Mar 2019 18:00:36 GMT):
so you are not supposed to use it
guoger (Wed, 13 Mar 2019 18:08:32 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=czewvLvva8FhyzD4n) @yacovm it's actually overwritten here: BlockMetadataIndex_LAST_CONFIG
guoger (Wed, 13 Mar 2019 18:08:32 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=czewvLvva8FhyzD4n) @yacovm it's actually overwritten here: https://github.com/hyperledger/fabric/blob/dbb4451c44095af27c27fbe34aad1d028a982d67/orderer/common/multichannel/blockwriter.go#L216-L221
yacovm (Wed, 13 Mar 2019 18:10:12 GMT):
damn. I somehow thought this writeBlock passes through directly to the ledger
yacovm (Wed, 13 Mar 2019 18:10:29 GMT):
so i need to use the chain's write block no?
yacovm (Wed, 13 Mar 2019 18:10:33 GMT):
@guoger @jyellick
jyellick (Wed, 13 Mar 2019 18:11:31 GMT):
https://github.com/hyperledger/fabric/blob/master/orderer/common/multichannel/blockwriter.go#L153-L178
jyellick (Wed, 13 Mar 2019 18:11:52 GMT):
It looks like we get lucky, in that if you supply `nil` as the metadata, then we do _not_ override the metadata which is passed in
jyellick (Wed, 13 Mar 2019 18:12:15 GMT):
However, the last config block metadata (stored in a different part of the block metadata) is always computed by the block writing.
jyellick (Wed, 13 Mar 2019 18:13:14 GMT):
Actually, I think we have bigger problems here
guoger (Wed, 13 Mar 2019 18:13:20 GMT):
not `chain` method, but just `support.WriteConfigBlock` i think?
jyellick (Wed, 13 Mar 2019 18:13:26 GMT):
It looks to me like this would cause the receiving node to re-sign the block, no?
yacovm (Wed, 13 Mar 2019 18:13:36 GMT):
it will
jyellick (Wed, 13 Mar 2019 18:13:42 GMT):
Which seems like a pretty significant problem?
jyellick (Wed, 13 Mar 2019 18:13:50 GMT):
How can you ever onboard an orderer from a new org.
yacovm (Wed, 13 Mar 2019 18:13:51 GMT):
oh yeah
yacovm (Wed, 13 Mar 2019 18:14:04 GMT):
the problem is only at the tip
yacovm (Wed, 13 Mar 2019 18:14:09 GMT):
or actually
yacovm (Wed, 13 Mar 2019 18:14:15 GMT):
the entire interval
yacovm (Wed, 13 Mar 2019 18:14:17 GMT):
from the eviction block
yacovm (Wed, 13 Mar 2019 18:14:22 GMT):
to the last config block
jyellick (Wed, 13 Mar 2019 18:14:29 GMT):
The leader should generate the block metadata, include it in the block he disseminates, and all followers should blindly write it (I would think)
yacovm (Wed, 13 Mar 2019 18:14:46 GMT):
we need to just commit the block to the ledger without touching it, like the onboarding code
jyellick (Wed, 13 Mar 2019 18:14:59 GMT):
Yes
jyellick (Wed, 13 Mar 2019 18:15:38 GMT):
But I would argue this should be the case for non-catchup cases as well. We need to apply the config updates as we commit them, but all of the metadata manipulation should not be done by the follower.
yacovm (Wed, 13 Mar 2019 18:16:29 GMT):
what are non catchup cases?
yacovm (Wed, 13 Mar 2019 18:16:48 GMT):
regular commit of the block while you are in the channel?
jyellick (Wed, 13 Mar 2019 18:16:51 GMT):
Yes
yacovm (Wed, 13 Mar 2019 18:17:06 GMT):
I don't follow now....
yacovm (Wed, 13 Mar 2019 18:17:25 GMT):
> We need to apply the config updates as we commit them, but all of the metadata manipulation should not be done by the follower.
I thought we use `WriteConfigBlock`
yacovm (Wed, 13 Mar 2019 18:17:30 GMT):
which does the book-keeping?
jyellick (Wed, 13 Mar 2019 18:19:03 GMT):
Okay, so... let me simply describe how I think the flow should work:
1) Leader cuts a block.
2) Leader constructs the block metadata, including the consenter metadata, and the last config block index, and signs both of them, attaching them to the block.
3) Follower gets fully constructed block, including metadata. It blindly writes this block to its ledger.
4) If the block was a config block, it applies the embedded config immediately before the write to disk.
jyellick (Wed, 13 Mar 2019 18:20:03 GMT):
Today, my understanding is that we:
1) Leader cuts the block.
2) Follower/leader gets the block with only data.
3) Follower/leader constructs the block metadata, each signing the block.
4) Follower/leader signs the metadata and commits the config update just before committing it to disk.
yacovm (Wed, 13 Mar 2019 18:25:11 GMT):
but why do you think there is deviation in the metadata, @jyellick ?
jyellick (Wed, 13 Mar 2019 18:25:31 GMT):
What do you mean?
jyellick (Wed, 13 Mar 2019 18:25:58 GMT):
There is a deviation, unquestionably. The metadata signer and signature is different on each orderer.
yacovm (Wed, 13 Mar 2019 18:26:20 GMT):
the signature is different because of the randomness
yacovm (Wed, 13 Mar 2019 18:26:25 GMT):
but what aside of it is different?
jyellick (Wed, 13 Mar 2019 18:26:28 GMT):
The signer
yacovm (Wed, 13 Mar 2019 18:26:34 GMT):
ok, the identity
yacovm (Wed, 13 Mar 2019 18:26:37 GMT):
what else?
jyellick (Wed, 13 Mar 2019 18:26:38 GMT):
The identity is not necessarily authorized to sign that block
yacovm (Wed, 13 Mar 2019 18:26:43 GMT):
how come?
jyellick (Wed, 13 Mar 2019 18:26:46 GMT):
Especially in the case of a late org join/add
yacovm (Wed, 13 Mar 2019 18:26:51 GMT):
that's what I am missing
yacovm (Wed, 13 Mar 2019 18:26:57 GMT):
in a normal non eviction scenario
yacovm (Wed, 13 Mar 2019 18:27:01 GMT):
how come you're not authorized?
jyellick (Wed, 13 Mar 2019 18:27:21 GMT):
We start an ordering network with Jay, Yacov, and Jason. We process a few thousand blocks.
jyellick (Wed, 13 Mar 2019 18:27:29 GMT):
At block 10k, we add Artem.
jyellick (Wed, 13 Mar 2019 18:27:40 GMT):
For blocks 0-10k, Artem is _not_ an authorized signer for those blocks.
yacovm (Wed, 13 Mar 2019 18:27:41 GMT):
but Artem doesn't sign the blocks himself
yacovm (Wed, 13 Mar 2019 18:27:45 GMT):
obviously...
yacovm (Wed, 13 Mar 2019 18:27:54 GMT):
he pulls them and commits them into the ledger
yacovm (Wed, 13 Mar 2019 18:28:04 GMT):
that's what onboarding is doing
jyellick (Wed, 13 Mar 2019 18:28:09 GMT):
But if Artem writes those blocks to disk using `WriteBlock` or` WriteConfigBlock`, he re-signs those blocks as him.
yacovm (Wed, 13 Mar 2019 18:28:13 GMT):
but he does not
jyellick (Wed, 13 Mar 2019 18:28:31 GMT):
Okay, so that's good.
jyellick (Wed, 13 Mar 2019 18:29:02 GMT):
I'm still not sure we have all of the corner cases covered
yacovm (Wed, 13 Mar 2019 18:29:20 GMT):
```
// Replicator replicates chains
type Replicator struct {
DoNotPanicIfClusterNotReachable bool
Filter ChannelPredicate
SystemChannel string
ChannelLister ChannelLister
Logger *flogging.FabricLogger
Puller *BlockPuller
BootBlock *common.Block
AmIPartOfChannel SelfMembershipPredicate
LedgerFactory LedgerFactory
}
```
yacovm (Wed, 13 Mar 2019 18:29:29 GMT):
this is the struct that replicates blocks
yacovm (Wed, 13 Mar 2019 18:29:35 GMT):
it uses a ledger...
jyellick (Wed, 13 Mar 2019 18:30:14 GMT):
https://github.com/hyperledger/fabric/blob/master/orderer/consensus/etcdraft/chain.go#L912-L964
yacovm (Wed, 13 Mar 2019 18:30:15 GMT):
> I'm still not sure we have all of the corner cases covered
ok, can you give an example? you had me worried there for a bit
yacovm (Wed, 13 Mar 2019 18:30:26 GMT):
this catchup is once you're in the channel
jyellick (Wed, 13 Mar 2019 18:30:57 GMT):
So these blocks you are getting
jyellick (Wed, 13 Mar 2019 18:31:01 GMT):
Do they have metadata populated?
yacovm (Wed, 13 Mar 2019 18:31:13 GMT):
yeah
yacovm (Wed, 13 Mar 2019 18:31:20 GMT):
the etcdraft catchup is via Deliver API
yacovm (Wed, 13 Mar 2019 18:31:31 GMT):
so it is the same block from the disk
jyellick (Wed, 13 Mar 2019 18:31:39 GMT):
Okay, cool. Why do we not use `WriteConfigBlock` on config block here?
yacovm (Wed, 13 Mar 2019 18:31:48 GMT):
we do not? :thinking_face:
jyellick (Wed, 13 Mar 2019 18:32:02 GMT):
Oh... I see we do
yacovm (Wed, 13 Mar 2019 18:32:02 GMT):
we do... i think
jyellick (Wed, 13 Mar 2019 18:32:08 GMT):
I thought Jay had said we do not
yacovm (Wed, 13 Mar 2019 18:32:25 GMT):
we do not at replicating the blocks once you are out of the channel
yacovm (Wed, 13 Mar 2019 18:32:32 GMT):
i will fix it... shortly
jyellick (Wed, 13 Mar 2019 18:32:40 GMT):
Okay, so this is truly only an eviction case.
yacovm (Wed, 13 Mar 2019 18:32:47 GMT):
i guess i need to add an `AppendBlock` method to support, right?
jyellick (Wed, 13 Mar 2019 18:34:03 GMT):
So as not to corrupt the existing metadata. Yes, I think this makes senes.
jyellick (Wed, 13 Mar 2019 18:34:03 GMT):
So as not to corrupt the existing metadata. Yes, I think this makes sense.
yacovm (Wed, 13 Mar 2019 18:34:39 GMT):
I am really wondering why we didn't just have the leader sign the block before it proposed to to Raft
jyellick (Wed, 13 Mar 2019 18:34:42 GMT):
It sounds like it is not critical for v1.4.1, but I would still like for us to go back and clean things up such that metadata is not computed per consenter, but is pre-computed and signed by the leader only.
yacovm (Wed, 13 Mar 2019 18:34:44 GMT):
it needs to sign anyway
yacovm (Wed, 13 Mar 2019 18:34:53 GMT):
and this way you can reduce CPU load on followers
yacovm (Wed, 13 Mar 2019 18:34:58 GMT):
by having them not need to sign anything
jyellick (Wed, 13 Mar 2019 18:35:01 GMT):
Yes, this seems like the obvious choice to me
jyellick (Wed, 13 Mar 2019 18:35:20 GMT):
And you do not risk divergence in metadata in case of non-deterministic bug.
yacovm (Wed, 13 Mar 2019 18:35:26 GMT):
yeah
jyellick (Wed, 13 Mar 2019 18:36:26 GMT):
Okay... but this sounds too invasive for v1.4.1 unless there is a legitimate bug/hole we have not covered.
yacovm (Wed, 13 Mar 2019 18:36:42 GMT):
it's too invasive because of the UTs
yacovm (Wed, 13 Mar 2019 19:16:32 GMT):
https://github.com/hyperledger/fabric/blob/release-1.4/sampleconfig/configtx.yaml#L341
yacovm (Wed, 13 Mar 2019 19:16:43 GMT):
we are keeping 100MB per channel in memory right @guoger ?
yacovm (Wed, 13 Mar 2019 21:44:18 GMT):
if we have lots of channels, this might become a problem and the orderers can run out of memory and then there is no way of backing off from this without doing a config transaction. I guess if the orderes are restarted then this isn't filled, and we can then do a config transaction before this gets filled up to 100MB again, right?
guoger (Wed, 13 Mar 2019 23:32:52 GMT):
When node starts, it loads all data since last snapshot, which could be 0-100 MB. We can surely reduce this number as part of my CR that tunes parameters
guoger (Wed, 13 Mar 2019 23:35:21 GMT):
Signing blocks at leader before consenting is not hard, although we need to infer Raft Index and put it into metadata in this case
yacovm (Wed, 13 Mar 2019 23:41:50 GMT):
@guoger it's not only the signing, it's also the last config block
yacovm (Wed, 13 Mar 2019 23:41:50 GMT):
@guoger it's not only the signing, it's also the last config block index
yacovm (Wed, 13 Mar 2019 23:42:05 GMT):
i don't think this can be contained in v1.4.1 :(
yacovm (Wed, 13 Mar 2019 23:42:05 GMT):
i don't think this can be contained in v1.4.1
yacovm (Wed, 13 Mar 2019 23:42:20 GMT):
but in 2.0 it's possible....
guoger (Thu, 14 Mar 2019 01:20:44 GMT):
maybe it's hard to be contained in 1.4.1, but i'm trying to find difficulties here. basically with blockcreator being separated from blockwriter, we are not really dependent on the update of block info in blockwriter, e.g. prev_hash, block number, last config block number, etc, instead they are simply stored in blockcreator. Those info are actually loaded from ledger only when a node is just elected to be leader.
guoger (Thu, 14 Mar 2019 04:28:15 GMT):
does `counterfeiter` have official release yet?
jyellick (Thu, 14 Mar 2019 04:29:13 GMT):
Not that I'm aware of
guoger (Thu, 14 Mar 2019 04:30:17 GMT):
just checked... it's already v6.0.1 ... although the cli doesn't give any version info.. odd
jyellick (Thu, 14 Mar 2019 04:52:59 GMT):
The github page also explicitly mentions not wanting to 'complicate the CLI interface"
jyellick (Thu, 14 Mar 2019 04:53:11 GMT):
But it is crazy to me that there is not a version number
jyellick (Thu, 14 Mar 2019 04:53:38 GMT):
(Though dealing with fakes, and counterfeiter version weirdness, beats dealing with mocks most times)
guoger (Thu, 14 Mar 2019 04:58:29 GMT):
haha ya.. and we should transform other pkgs to start using fakes, e.g. blockwriter
guoger (Thu, 14 Mar 2019 05:01:12 GMT):
also, @jyellick do you mind rebasing your CR on top of @yacovm 's fix, so we can assert if it actually fixes the problem
jyellick (Thu, 14 Mar 2019 05:01:36 GMT):
Did I not merge his fix and rebase onto release-1.4 or am I missing one?
guoger (Thu, 14 Mar 2019 05:02:24 GMT):
ah, i have outdated information
guoger (Thu, 14 Mar 2019 05:02:39 GMT):
just noticed it's merged
jyellick (Thu, 14 Mar 2019 05:02:58 GMT):
The top of my stack passed CI so I thought the middle CI failure was likely a flake
jyellick (Thu, 14 Mar 2019 05:03:05 GMT):
(Hence re-running integration)
jyellick (Thu, 14 Mar 2019 06:27:16 GMT):
@guoger Looks like the fix from @yacovm doesn't solve my integration test problems entirely. I haven't really had a chance to analyze, if someone else has the opportunity great, otherwise I'll pick it up in the morning (but it is getting late EDT so I need to stop for the night)
guoger (Thu, 14 Mar 2019 06:27:45 GMT):
it's indeed very late... i've just opened that failure log
guoger (Thu, 14 Mar 2019 06:28:10 GMT):
however, CI is green for https://gerrit.hyperledger.org/r/c/30051/4
guoger (Thu, 14 Mar 2019 06:28:12 GMT):
@jyellick
jyellick (Thu, 14 Mar 2019 06:28:26 GMT):
Yes, I thought that was odd. Perhaps it is some logic bug I introduced and then fixed
guoger (Thu, 14 Mar 2019 06:29:28 GMT):
i'll take a look
jyellick (Thu, 14 Mar 2019 06:32:32 GMT):
Great, thanks, if you do not have time, no worries, but I will definitely appreciate it if you figure it out
guoger (Thu, 14 Mar 2019 07:08:09 GMT):
commented on the CR. just one line missing :P @jyellick
guoger (Thu, 14 Mar 2019 07:17:55 GMT):
question
- does orderer actually sign block metadata?
- why is metadata nested in another metadata in block proto?
yacovm (Thu, 14 Mar 2019 08:40:43 GMT):
Of course it signs it
yacovm (Thu, 14 Mar 2019 08:41:10 GMT):
Otherwise a peer can change it
guoger (Thu, 14 Mar 2019 08:44:10 GMT):
could you point me to the code? thx!
guoger (Thu, 14 Mar 2019 08:44:10 GMT):
could you point me to the code? thx! @yacovm
yacovm (Thu, 14 Mar 2019 09:04:51 GMT):
the code that does what? the signinig? @guoger
guoger (Thu, 14 Mar 2019 09:05:29 GMT):
yes
guoger (Thu, 14 Mar 2019 09:05:50 GMT):
where is block metadata signed
yacovm (Thu, 14 Mar 2019 09:06:49 GMT):
```
// commitBlock should only ever be invoked with the bw.committingBlock held
// this ensures that the encoded config sequence numbers stay in sync
func (bw *BlockWriter) commitBlock(encodedMetadataValue []byte) {
// Set the orderer-related metadata field
if encodedMetadataValue != nil {
bw.lastBlock.Metadata.Metadata[cb.BlockMetadataIndex_ORDERER] = protoutil.MarshalOrPanic(&cb.Metadata{Value: encodedMetadataValue})
}
bw.addBlockSignature(bw.lastBlock)
bw.addLastConfigSignature(bw.lastBlock)
err := bw.support.Append(bw.lastBlock)
if err != nil {
logger.Panicf("[channel: %s] Could not append block: %s", bw.support.ChainID(), err)
}
logger.Debugf("[channel: %s] Wrote block %d", bw.support.ChainID(), bw.lastBlock.GetHeader().Number)
}
func (bw *BlockWriter) addBlockSignature(block *cb.Block) {
blockSignature := &cb.MetadataSignature{
SignatureHeader: protoutil.MarshalOrPanic(protoutil.NewSignatureHeaderOrPanic(bw.support)),
}
// Note, this value is intentionally nil, as this metadata is only about the signature, there is no additional metadata
// information required beyond the fact that the metadata item is signed.
blockSignatureValue := []byte(nil)
blockSignature.Signature = protoutil.SignOrPanic(
bw.support,
util.ConcatenateBytes(blockSignatureValue, blockSignature.SignatureHeader, protoutil.BlockHeaderBytes(block.Header)),
)
block.Metadata.Metadata[cb.BlockMetadataIndex_SIGNATURES] = protoutil.MarshalOrPanic(&cb.Metadata{
Value: blockSignatureValue,
Signatures: []*cb.MetadataSignature{
blockSignature,
},
})
}
```
guoger (Thu, 14 Mar 2019 09:06:58 GMT):
i only see these two
```
bw.addBlockSignature(bw.lastBlock)
bw.addLastConfigSignature(bw.lastBlock)
```
yacovm (Thu, 14 Mar 2019 09:07:25 GMT):
I think the metadata of the raft is not signed
yacovm (Thu, 14 Mar 2019 09:08:04 GMT):
i guess maybe it's ok for CFT because we use TLS ?
jyellick (Thu, 14 Mar 2019 14:58:46 GMT):
The stack ending in https://gerrit.hyperledger.org/r/c/30051/ has passed CI and should be ready for review
guoger (Fri, 15 Mar 2019 01:29:44 GMT):
re FAB-14648: yeah i think we should definitely validate consenter data when config tx comes in. I'll take on this. However, I'm curious why you also proposed to halt the chain as an alternative? After validation is added, we should still panic on `configureComm`, because that error, in theory, should never happen, right? @yacovm
guoger (Fri, 15 Mar 2019 01:29:44 GMT):
re FAB-14648: yeah i think we should definitely validate consenter data when config tx comes in. I'll take on this. However, I'm curious why you also proposed to halt the chain as an alternative? *After validation is added*, we should still panic on `configureComm`, because that error, in theory, should never happen, right? @yacovm
jyellick (Fri, 15 Mar 2019 01:30:53 GMT):
+1 if we validate inputs, we shouldn't be able to have a committed block with bad config. We panic in other parts of the code over such a situation.
jyellick (Fri, 15 Mar 2019 01:30:53 GMT):
+1. If we validate inputs, we shouldn't be able to have a committed block with bad config. We panic in other parts of the code over such a situation.
C0rWin (Fri, 15 Mar 2019 01:35:54 GMT):
isn't it a problem where wrong update config tx cause to entire cluster of OSNs to panic?
C0rWin (Fri, 15 Mar 2019 01:39:23 GMT):
I can understand why there is a need to panic to prevent fork in the state or divergence, but in that case Halting misconfigured chain would be just fine, no? I mean OSN serving several chains, hence misconfiguration of one should not prevent those OSNs from serving the rest.
jyellick (Fri, 15 Mar 2019 01:40:17 GMT):
Well, we would panic on situations we should not encounter. For instance, non-validly encoded X.509 certs.
jyellick (Fri, 15 Mar 2019 01:40:35 GMT):
This would indicate that this block was committed erroneously, and all bets are off. Safer to panic and identify the underlying cause.
jyellick (Fri, 15 Mar 2019 01:41:16 GMT):
(Assuming we valid the inputs before commit, we should not encounter invalid data reading committed data)
jyellick (Fri, 15 Mar 2019 01:41:16 GMT):
(Assuming we valid the inputs before commit, we should not encounter invalid data reading committed state)
C0rWin (Fri, 15 Mar 2019 01:41:36 GMT):
so if I'd like to take OSNs cluster down, assuming malicious admin all I need is to musconfigure X.509 for one chain
C0rWin (Fri, 15 Mar 2019 01:42:54 GMT):
I can just create a new one organization channel w/ wrong cert and cause panic on all OSNs
C0rWin (Fri, 15 Mar 2019 01:43:16 GMT):
sounds a bit risky to me, unless I'm missing something
C0rWin (Fri, 15 Mar 2019 01:47:01 GMT):
IMO we need to add sanity check for config transaction to check for certs validity and reject config updates where we have a problem w/ cert
jyellick (Fri, 15 Mar 2019 01:47:02 GMT):
If you are a malicious admin
jyellick (Fri, 15 Mar 2019 01:47:11 GMT):
And you control a Raft node
jyellick (Fri, 15 Mar 2019 01:47:37 GMT):
You may force that node to be leader, and you may commit blocks with say, a capability that does not exist.
jyellick (Fri, 15 Mar 2019 01:47:49 GMT):
You will cause all nodes in the network to panic.
jyellick (Fri, 15 Mar 2019 01:48:46 GMT):
If a block is committed with a signature satisfying the block validation policy. Then that block is assumed to be valid. If that's not true, then a myriad of attacks become possible.
C0rWin (Fri, 15 Mar 2019 01:49:18 GMT):
k, yes you are right... got it. but isn't simplier then just not to accept transaction with misconfigured X.509 cert?
C0rWin (Fri, 15 Mar 2019 01:49:40 GMT):
then it make sense to panic
jyellick (Fri, 15 Mar 2019 01:49:42 GMT):
Ah yes, 100% we need to do that. That is Jay's suggestion above.
jyellick (Fri, 15 Mar 2019 01:50:32 GMT):
If we validate the update to contain a valid consenter set, then no need to worry about halting vs. panic-ing. Because we cannot encounter a committed config with bad consenterss
jyellick (Fri, 15 Mar 2019 01:50:32 GMT):
If we validate the update to contain a valid consenter set, then no need to worry about halting vs. panic-ing. Because we cannot encounter a committed config with bad consenter set.
C0rWin (Fri, 15 Mar 2019 01:50:42 GMT):
having validation in place, makes more sense to have panic in that code path
C0rWin (Fri, 15 Mar 2019 01:51:20 GMT):
I just misread the part where Jay proposed to add validation
guoger (Fri, 15 Mar 2019 02:30:34 GMT):
late to the party... re-edit my message to mark that part bold
guoger (Fri, 15 Mar 2019 02:30:34 GMT):
late to the party... re-edited my message to mark that part bold
guoger (Fri, 15 Mar 2019 02:44:37 GMT):
btw, why do we call it `ConfigMetadata` instead of just `Config`? @jyellick
jyellick (Fri, 15 Mar 2019 02:45:36 GMT):
@guoger It's stored in the `metadata` field of the channel config's `ConsensusType` value
jyellick (Fri, 15 Mar 2019 02:46:07 GMT):
Most appropriately it is the "Channel config consensus type metadata", but that seemed a bit long
jyellick (Fri, 15 Mar 2019 02:46:42 GMT):
I can see how `ConfigMetadata` would read as "Metadata about the config", but I meant it to be "metadata from the config"
guoger (Fri, 15 Mar 2019 02:47:13 GMT):
i think it's more of "config from the metadata" though?
guoger (Fri, 15 Mar 2019 02:47:35 GMT):
i would expect the statement to be "consensustype metadata stores etcdraft configurations"
guoger (Fri, 15 Mar 2019 02:48:11 GMT):
the additional "metadata" sounds a bit repetitive
jyellick (Fri, 15 Mar 2019 02:50:21 GMT):
Do you not think calling it plain `Config` would be a bit misleading, as it is only a part of the configuration?
guoger (Fri, 15 Mar 2019 02:54:10 GMT):
i don't have strong opinions on this, but i feel metadata is still somewhat more confusing because "metadata" doesn't really convey the message "this field contains a subset of configurations for etcdraft consensus"
guoger (Fri, 15 Mar 2019 02:55:19 GMT):
i certainly could live with it as i've been reading it for a while :)
jyellick (Fri, 15 Mar 2019 02:57:17 GMT):
I don't love the name, and if we have a great replacement for it I'm all ears. I thought `Metadata` and `RaftMetadata` were quite confusing, which is why I moved them to `ConfigMetadata` and `BlockMetadata`, but.... I won't claim they're perfect, just better.
guoger (Fri, 15 Mar 2019 03:16:13 GMT):
sure. I was trying to explore if this can be more clear, since we are editing it anyway.
jyellick (Fri, 15 Mar 2019 03:16:52 GMT):
I prefer `ConfigMetadata` over `Config` personally, I just think we've overloaded that word too much. But maybe there's some other better option out there.
jyellick (Fri, 15 Mar 2019 03:16:52 GMT):
I prefer `ConfigMetadata` over `Config` personally, I just think we've overloaded that word (Config) too much not to have some other descriptor attached. But maybe there's some other better option out there.
jyellick (Fri, 15 Mar 2019 15:56:28 GMT):
@yacovm @C0rWin https://gerrit.hyperledger.org/r/c/30051/ and Jay's CRs above could use some attention as fixes for the v1.4 stream
guoger (Mon, 18 Mar 2019 13:32:24 GMT):
what attack scenario is it protecting against by hardening tls? @yacovm
yacovm (Mon, 18 Mar 2019 13:34:18 GMT):
so, let's say we have an orderer org and somehow i get a certificate of an orderer node that has the same host
yacovm (Mon, 18 Mar 2019 13:34:42 GMT):
then i can do a network spoofing and send you bad blocks
yacovm (Mon, 18 Mar 2019 13:34:45 GMT):
and fork the network
yacovm (Mon, 18 Mar 2019 13:34:57 GMT):
however if we use TLS pinning, then I actually have to steal the private key of the orderer node
yacovm (Mon, 18 Mar 2019 13:35:10 GMT):
which means i need to actually hack inside a real orderer
yacovm (Mon, 18 Mar 2019 13:35:13 GMT):
so it's harder
guoger (Mon, 18 Mar 2019 13:49:24 GMT):
ah, so you were saying we should enforce tls pinning for replicator?
yacovm (Mon, 18 Mar 2019 14:07:04 GMT):
that's what i was proposing
yacovm (Mon, 18 Mar 2019 14:07:11 GMT):
but Jason was right in the point of the different listener
yacovm (Mon, 18 Mar 2019 14:07:13 GMT):
I forgot that part
yacovm (Mon, 18 Mar 2019 14:07:19 GMT):
even though I implemented it :joy:
jyellick (Mon, 18 Mar 2019 17:59:14 GMT):
So what do we propose as a solution? Certainly we can enforce that the TLS cert is from an orderer, but it's not a particularly strong guarantee. To be fair though, our default is that the block is signed by 'some ordering org identity', so I'm not sure it is much weaker.
jyellick (Mon, 18 Mar 2019 17:59:37 GMT):
(We should perhaps look into changing this block validation policy to be more specific as well)
yacovm (Mon, 18 Mar 2019 18:11:18 GMT):
so @jyellick can you perhaps explain more thoroughly what you had in mind with the signature check? you said we can extend a protobuf field, which is it?
yacovm (Mon, 18 Mar 2019 18:11:30 GMT):
```
func (bw *BlockWriter) addBlockSignature(block *cb.Block) {
blockSignature := &cb.MetadataSignature{
SignatureHeader: protoutil.MarshalOrPanic(protoutil.NewSignatureHeaderOrPanic(bw.support)),
}
// Note, this value is intentionally nil, as this metadata is only about the signature, there is no additional metadata
// information required beyond the fact that the metadata item is signed.
blockSignatureValue := []byte(nil)
blockSignature.Signature = protoutil.SignOrPanic(
bw.support,
util.ConcatenateBytes(blockSignatureValue, blockSignature.SignatureHeader, protoutil.BlockHeaderBytes(block.Header)),
)
block.Metadata.Metadata[cb.BlockMetadataIndex_SIGNATURES] = protoutil.MarshalOrPanic(&cb.Metadata{
Value: blockSignatureValue,
Signatures: []*cb.MetadataSignature{
blockSignature,
},
})
}
```
yacovm (Mon, 18 Mar 2019 18:12:22 GMT):
the signature is over the the `SignatureHeader` bytes which is "common" and I guess we don't want to change that, and also over the `BlockHeader` which is.... also common
yacovm (Mon, 18 Mar 2019 18:12:26 GMT):
what can we do?
jyellick (Mon, 18 Mar 2019 18:15:51 GMT):
`blockSignatureValue` is always empty today... we could shove the metadata in there
jyellick (Mon, 18 Mar 2019 18:15:59 GMT):
The downstream code does not check that it is empty
jyellick (Mon, 18 Mar 2019 18:15:59 GMT):
The downstream code does not check that it is empty / use that value at all
yacovm (Mon, 18 Mar 2019 18:25:10 GMT):
so we make a protobuf message for the metadata value.... and we put inside it, the metadata?
yacovm (Mon, 18 Mar 2019 18:25:18 GMT):
why not put the entire block metadata in it?
yacovm (Mon, 18 Mar 2019 18:25:21 GMT):
well, the hash of it
yacovm (Mon, 18 Mar 2019 18:25:39 GMT):
this way we also can verify the last config block
yacovm (Mon, 18 Mar 2019 18:25:46 GMT):
ah wait the metadata is a protobuf
yacovm (Mon, 18 Mar 2019 18:25:48 GMT):
not bytes
jyellick (Mon, 18 Mar 2019 18:26:08 GMT):
The metadata proto is just a simple wrapper
jyellick (Mon, 18 Mar 2019 18:26:20 GMT):
It allows you to sign over aribtrary bytes
yacovm (Mon, 18 Mar 2019 18:26:20 GMT):
```
type BlockMetadata struct {
Metadata [][]byte
```
yacovm (Mon, 18 Mar 2019 18:26:34 GMT):
we can put the hash of this byte slice slice
yacovm (Mon, 18 Mar 2019 18:26:43 GMT):
in the blockSignatureValue
yacovm (Mon, 18 Mar 2019 18:26:43 GMT):
no?
jyellick (Mon, 18 Mar 2019 18:26:48 GMT):
No
jyellick (Mon, 18 Mar 2019 18:26:59 GMT):
Because the peer later mutates this slice
yacovm (Mon, 18 Mar 2019 18:27:20 GMT):
ah right the invalid Txns
yacovm (Mon, 18 Mar 2019 18:27:28 GMT):
ok so just the ones that it doesn't mutate then
jyellick (Mon, 18 Mar 2019 18:27:57 GMT):
I don't see why we should worry about hashes
jyellick (Mon, 18 Mar 2019 18:28:12 GMT):
Why not simply put the metadata directly into the first nil bytes which is already signed over.
yacovm (Mon, 18 Mar 2019 18:28:17 GMT):
```
message BlockSignatureDigests {
repeated bytes metadatas
}
```
something like this?
jyellick (Mon, 18 Mar 2019 18:28:31 GMT):
One minute while I type it up...
yacovm (Mon, 18 Mar 2019 18:29:12 GMT):
I think whatever replaces the `nil` should be a marshaled protobuf message for extendability
jyellick (Mon, 18 Mar 2019 18:34:18 GMT):
What we have today:
```&common.BlockMetadata{
utils.MarshalOrPanic(&common.BlockMetadata{
Value: nil,
Signatures: ...,
}),
utils.MarshalOrPanic(&common.BlockMetadata{
Value: utils.MarshalOrPanic(&cb.LastConfig{Index: bw.lastConfigBlockNum}),
Signatures: ...,
}),
nil,
opaqueConsenterMetadata,
}
```
This could/should become:
```&common.BlockMetadata{
utils.MarshalOrPanic(&common.BlockMetadata{
Value: utils.MarshalOrPanic(&cb.OrdererBlockMetadata{
LastConfigIndex: bw.lastConfigBlockNum,
OpaqueConsenterMetadata: opaqueConsenterMetadata,
}),
Signatures: ...,
}),
copy(index at 0), -- long term plan to nil this once clients are updated.
nil,
nil,
}
```
jyellick (Mon, 18 Mar 2019 18:34:18 GMT):
What we have today:
```&common.BlockMetadata{
Metadata: [][]byte{
utils.MarshalOrPanic(&common.BlockMetadata{
Value: nil,
Signatures: ...,
}),
utils.MarshalOrPanic(&common.BlockMetadata{
Value: utils.MarshalOrPanic(&cb.LastConfig{Index: bw.lastConfigBlockNum}),
Signatures: ...,
}),
nil,
opaqueConsenterMetadata,
},
}
```
This could/should become:
```&common.BlockMetadata{
Metadata: [][]byte{
utils.MarshalOrPanic(&common.BlockMetadata{
Value: utils.MarshalOrPanic(&cb.OrdererBlockMetadata{
LastConfigIndex: bw.lastConfigBlockNum,
OpaqueConsenterMetadata: opaqueConsenterMetadata,
}),
Signatures: ...,
}),
copy(index at 0), -- long term plan to nil this once clients are updated.
nil,
nil,
},
}
```
jyellick (Mon, 18 Mar 2019 18:34:18 GMT):
What we have today:
```&common.BlockMetadata{
Metadata: [][]byte{
utils.MarshalOrPanic(&common.Metadata{
Value: nil,
Signatures: ...,
}),
utils.MarshalOrPanic(&common.Metadata{
Value: utils.MarshalOrPanic(&cb.LastConfig{Index: bw.lastConfigBlockNum}),
Signatures: ...,
}),
nil,
opaqueConsenterMetadata,
},
}
```
This could/should become:
```&common.BlockMetadata{
Metadata: [][]byte{
utils.MarshalOrPanic(&common.Metadata{
Value: utils.MarshalOrPanic(&cb.OrdererBlockMetadata{
LastConfigIndex: bw.lastConfigBlockNum,
OpaqueConsenterMetadata: opaqueConsenterMetadata,
}),
Signatures: ...,
}),
copy(index at 0), -- long term plan to nil this once clients are updated.
nil,
nil,
},
}
```
jyellick (Mon, 18 Mar 2019 18:35:13 GMT):
In the event that an orderer starts up, it would first check to see if metadata[0].Value is non-nil. If so, then take the consenter metadata from there. Otherwise, fall back to index 3.
jyellick (Mon, 18 Mar 2019 18:36:06 GMT):
If we did the proto fields right, we could simply duplicate that first metadata for the second, and eliminate that second signature.
yacovm (Mon, 18 Mar 2019 18:39:25 GMT):
@jyellick I think you're not accurate... the `BlockMetadata` is a struct with a `Metadata` field which is just a byte slice slice and each byte slice is its own protobuf, not another `BlockMetadata`
yacovm (Mon, 18 Mar 2019 18:39:25 GMT):
@jyellick I think you're not being accurate... the `BlockMetadata` is a struct with a `Metadata` field which is just a byte slice slice and each byte slice is its own protobuf, not another `BlockMetadata`
jyellick (Mon, 18 Mar 2019 18:41:10 GMT):
Sorry... I was a bit sloppy
jyellick (Mon, 18 Mar 2019 18:41:17 GMT):
Let me fix...
jyellick (Mon, 18 Mar 2019 18:42:29 GMT):
Fixed
yacovm (Mon, 18 Mar 2019 18:43:09 GMT):
I still don't think we have a double nesting of `BlockMetadata`
yacovm (Mon, 18 Mar 2019 18:43:18 GMT):
```
metadata.Metadata[common.BlockMetadataIndex_LAST_CONFIG] = protoutil.MarshalOrPanic(&common.LastConfig{
Index: 0,
})
```
yacovm (Mon, 18 Mar 2019 18:43:41 GMT):
this is an example
jyellick (Mon, 18 Mar 2019 18:43:43 GMT):
``` block.Metadata.Metadata[cb.BlockMetadataIndex_LAST_CONFIG] = protoutil.MarshalOrPanic(&cb.Metadata{
Value: lastConfigValue,
Signatures: []*cb.MetadataSignature{
lastConfigSignature,
},
})
```
yacovm (Mon, 18 Mar 2019 18:44:30 GMT):
oh.... then I think we have a bug
yacovm (Mon, 18 Mar 2019 18:44:38 GMT):
:joy:
yacovm (Mon, 18 Mar 2019 18:45:19 GMT):
https://github.com/hyperledger/fabric/blob/master/orderer/common/cluster/replication.go#L622
jyellick (Mon, 18 Mar 2019 18:45:34 GMT):
https://github.com/hyperledger/fabric/blob/release-1.4/orderer/common/multichannel/blockwriter.go#L216-L221
jyellick (Mon, 18 Mar 2019 18:45:48 GMT):
A bug indeed!
yacovm (Mon, 18 Mar 2019 18:46:37 GMT):
but all our tests work.
yacovm (Mon, 18 Mar 2019 18:46:41 GMT):
i guess no one validates that
jyellick (Mon, 18 Mar 2019 18:46:51 GMT):
I would think marshaling would have problems
jyellick (Mon, 18 Mar 2019 18:46:51 GMT):
I would think un-marshaling would have problems
yacovm (Mon, 18 Mar 2019 18:47:02 GMT):
right, no one tries
yacovm (Mon, 18 Mar 2019 18:47:08 GMT):
no one, as in - no code
jyellick (Mon, 18 Mar 2019 18:47:10 GMT):
Ah
yacovm (Mon, 18 Mar 2019 18:47:21 GMT):
sigh, I'll open a JIRA and fix now
jyellick (Mon, 18 Mar 2019 18:48:01 GMT):
This would only be for the genesis block no?
jyellick (Mon, 18 Mar 2019 18:49:06 GMT):
I suspect that because peers are joined with this block, it bypasses many of the normal checks. For instance, it does not check the 'block signature' because there is no config to validate it against. There would similarly be no reason for it to check the last config block, as it must be this block.
jyellick (Mon, 18 Mar 2019 18:49:16 GMT):
(Just trying to account for why we wouldn't have noticed this in tests)
yacovm (Mon, 18 Mar 2019 18:49:19 GMT):
> In the event that an orderer starts up, it would first check to see if metadata[0].Value is non-nil. If so, then take the consenter metadata from there.
ok I understand now.
yacovm (Mon, 18 Mar 2019 18:49:35 GMT):
and this is backward compatible for peers too even if they don't check it
jyellick (Mon, 18 Mar 2019 18:49:41 GMT):
Exactly
yacovm (Mon, 18 Mar 2019 18:49:50 GMT):
why didn't we do it, in the 1st place?
yacovm (Mon, 18 Mar 2019 18:50:08 GMT):
when we added the additional last config
yacovm (Mon, 18 Mar 2019 18:50:51 GMT):
> This would only be for the genesis block no?
precisely. and the hash of the blocks onward isn't effected since the hash doesn't depend on the metadata
jyellick (Mon, 18 Mar 2019 18:50:58 GMT):
Right, so, basically, we didn't want to require the same 'validation policy' for 'last config' and for 'block validity'
yacovm (Mon, 18 Mar 2019 18:51:08 GMT):
and why is that?
jyellick (Mon, 18 Mar 2019 18:51:17 GMT):
In retrospect it was a mistake.
yacovm (Mon, 18 Mar 2019 18:51:27 GMT):
you thought peers might set it?
yacovm (Mon, 18 Mar 2019 18:51:43 GMT):
or any orderer on its own?
jyellick (Mon, 18 Mar 2019 18:51:43 GMT):
No, we thought that in the BFT case we might not want to require f+1 signatures on last config index
jyellick (Mon, 18 Mar 2019 18:51:50 GMT):
But would want it for block validity
jyellick (Mon, 18 Mar 2019 18:52:00 GMT):
This was during the time when gossip was going to speculatively share uncommitted blocks.
yacovm (Mon, 18 Mar 2019 18:52:23 GMT):
but then we went on and used the last config for other stuff in the peer, but not for gossip
jyellick (Mon, 18 Mar 2019 18:52:42 GMT):
Right... in all, I think we screwed up the metadata stuff pretty thoroughly.
jyellick (Mon, 18 Mar 2019 18:52:58 GMT):
But, we can consolidate the orderer stuff under this first field, and hopefully fix much of it.
jyellick (Mon, 18 Mar 2019 20:52:51 GMT):
@yacovm Are you planning to tackle this metadata consolidation thing, or would you like me to?
yacovm (Mon, 18 Mar 2019 21:00:48 GMT):
I can do it, sure
yacovm (Mon, 18 Mar 2019 21:01:03 GMT):
put it on my tab
yacovm (Mon, 18 Mar 2019 21:02:41 GMT):
https://jira.hyperledger.org/browse/FAB-14697
yacovm (Mon, 18 Mar 2019 21:03:00 GMT):
(will fill the description after I do it)
guoger (Tue, 19 Mar 2019 05:21:10 GMT):
@jyellick raised a concern:
> For reconfiguration, is it possible to go from 2 nodes to 1? I assume so... but I'm concerned about the TLS cert being evicted before we can get consensus on the reconfiguration.
and I think it's valid, and something i overlooked :( . raft protocol requires removed leader to continue coordinating the follower in this case. etcd itself (not raft lib) provides a config option `--strict-reconfig-check` to "Reject reconfiguration requests that would cause quorum loss" and i think we should do the same.
guoger (Tue, 19 Mar 2019 05:21:10 GMT):
@jyellick raised a concern:
> For reconfiguration, is it possible to go from 2 nodes to 1? I assume so... but I'm concerned about the TLS cert being evicted before we can get consensus on the reconfiguration.
and I think it's valid, and something i overlooked :( . raft protocol requires removed leader to continue coordinating the follower in this case.
(etcd itself (not raft lib) provides a config option `--strict-reconfig-check` to "Reject reconfiguration requests that would cause quorum loss")
guoger (Tue, 19 Mar 2019 05:53:32 GMT):
config is rejected if removal would cause quorum loss *and* removed node is currently leader
guoger (Tue, 19 Mar 2019 05:53:32 GMT):
config is rejected if removal would cause quorum loss *and* removed node is currently leader. Although I'm still seeing one caveat: we are doing two rounds of consensus for membership change, and if there's a failover in the middle, we fall back to the original problem.
guoger (Tue, 19 Mar 2019 05:53:32 GMT):
the problem is essentially when the removal affects quorum, the actual quorum size is *not* changed until this config change is applied on quorum, and communication should be maintained
guoger (Tue, 19 Mar 2019 05:53:32 GMT):
the problem is essentially when the removal affects quorum, the actual quorum size is *not* changed until this config change is applied on quorum, and communication should be maintained (this affects rotation as well)
guoger (Tue, 19 Mar 2019 09:22:21 GMT):
a short term solution would be to do something similar to cert rotation -- transfer leadership and delay `configureComm`. This does not entirely solve the problem though, in case of node crash and restarts at this point of time. A long term solution would be to alter communication, so that if a node finds itself *not* among consenter set, it still connects to other nodes, and the chain can be halted only if it finds any other node has higher height than itself. obviously other nodes would drop the connection to it once it's removed from their consenter set, and this is fine because cluster is ready to move on.
guoger (Tue, 19 Mar 2019 09:22:21 GMT):
Short term solution would be to do something similar to cert rotation -- transfer leadership and delay `configureComm`. This does not entirely solve the problem though, in case of node crash and restarts at this point of time.
Long term solution would be to alter communication, so that if a node finds itself *not* among consenter set, it still connects to other nodes, and the chain can be halted only if it finds any other node has higher height than itself. obviously other nodes would drop the connection to it once it's removed from their consenter set, and this is fine because cluster is ready to move on.
guoger (Tue, 19 Mar 2019 09:24:11 GMT):
wdyt @yacovm
yacovm (Tue, 19 Mar 2019 09:55:27 GMT):
I don't understand @guoger
yacovm (Tue, 19 Mar 2019 09:55:32 GMT):
> Long term solution would be to alter communication, so that if a node finds itself *not* among consenter set, it still connects to other nodes
what do you mean here
yacovm (Tue, 19 Mar 2019 09:55:41 GMT):
in the communication you are never aware of yourself
yacovm (Tue, 19 Mar 2019 09:55:44 GMT):
only of your other nodes
guoger (Tue, 19 Mar 2019 10:29:03 GMT):
Ah, so that part is no-op then. We just need to transfer leadership. Also chain should be started if last block evicts itself and it’s required to form quorum, in case other nodes haven’t picked this conf change
tock (Tue, 19 Mar 2019 12:19:33 GMT):
@jyellick @yacovm I divide the bi CR for abort & recovery (Which you both reviewed) into 3 smaller CRs.
- The first is just renaming structs/vars, with no functional change: https://gerrit.hyperledger.org/r/#/c/30213/
- The second adds the validation of config updates in the broadcast phase (with the tests that I promised to add later) https://gerrit.hyperledger.org/r/#/c/30216/
- The third contains the actual abort & recovery logic: https://gerrit.hyperledger.org/r/#/c/29710/
tock (Tue, 19 Mar 2019 12:19:33 GMT):
@jyellick @yacovm I divide the big CR for abort & recovery (Which you both reviewed) into 3 smaller CRs.
- The first is just renaming structs/vars, with no functional change: https://gerrit.hyperledger.org/r/#/c/30213/
- The second adds the validation of config updates in the broadcast phase (with the tests that I promised to add later) https://gerrit.hyperledger.org/r/#/c/30216/
- The third contains the actual abort & recovery logic: https://gerrit.hyperledger.org/r/#/c/29710/
tock (Tue, 19 Mar 2019 12:19:33 GMT):
@jyellick @yacovm I divide the big CR for abort & recovery (Which you both reviewed) into 3 smaller CRs.
- The first is just renaming structs/vars, with no functional change: https://gerrit.hyperledger.org/r/#/c/30213/
- The second adds the validation of config updates in the broadcast phase (with the tests that I promised to add later) https://gerrit.hyperledger.org/r/#/c/30216/
- The third contains the actual abort & recovery logic: https://gerrit.hyperledger.org/r/#/c/29710/
The third carries all comments and patched so far, the first two are new.
tock (Tue, 19 Mar 2019 12:22:46 GMT):
To all reviewers out there, @yacovm needs a helping hand in reviewing the 5-node BYFN sample: https://gerrit.hyperledger.org/r/#/c/29927/
yacovm (Tue, 19 Mar 2019 13:18:57 GMT):
I dontneed a hand, i +2ed
tock (Tue, 19 Mar 2019 13:32:37 GMT):
we need another +2 and merge it.
tock (Tue, 19 Mar 2019 13:33:59 GMT):
@yacovm Gossip is misbehaving in CI: https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/10666/console
tock (Tue, 19 Mar 2019 14:53:15 GMT):
Data Race in
16:05:08 FAIL github.com/hyperledger/fabric/orderer/common/cluster
https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/10667/console
jyellick (Wed, 20 Mar 2019 19:01:31 GMT):
@C0rWin or @yacovm could you please take a look at https://gerrit.hyperledger.org/r/c/29954/ and https://gerrit.hyperledger.org/r/c/30114/ to get reviewed and hopefully merged, they have been hanging out for a while and are holding up test.
yacovm (Wed, 20 Mar 2019 21:30:00 GMT):
I don't understand what's going on in the 2nd one :/
jyellick (Thu, 21 Mar 2019 03:52:25 GMT):
And for @yacovm or @C0rWin again https://gerrit.hyperledger.org/r/c/30177/ and below are all cherry-picks which have been merged on release-1.4, so should be an easy +2 and a merge.
guoger (Thu, 21 Mar 2019 07:21:27 GMT):
@yacovm when a node is *removed* from cluster, is it ok *not to* configure communication on remaining nodes? I'm asking because if leader configures communication too fast, the removed node might not be able to pick up this removal quick enough. Even though this node will be eventually suspected and evicted, but i'd like this to be more deterministic.
guoger (Thu, 21 Mar 2019 07:22:31 GMT):
in particular, if a node does not reconfigure communication to accommodate to removed node, would it cause any unexpected problems?
yacovm (Thu, 21 Mar 2019 07:55:11 GMT):
that's a question you need to ask yourself since communication simply provides you with support of tagging a message with the ID based on the certificate, but I can't think of any attack vector given we never reuse the same ID for different certificates @guoger .
guoger (Thu, 21 Mar 2019 08:17:03 GMT):
I see. I was simply concerned if it tries to establish connection in the background and falls into retry loop, since the remote might already exit
dave.enyeart (Fri, 22 Mar 2019 00:55:57 GMT):
@tock I merged byfn update to 5 raft orderers
dave.enyeart (Fri, 22 Mar 2019 00:56:21 GMT):
could you cherry pick each of the byfn raft updates to release-1.4 fabric-samples?
dave.enyeart (Fri, 22 Mar 2019 01:02:18 GMT):
BTW, why do we call it RAFT2 in byfn? https://github.com/hyperledger/fabric-samples/blob/master/first-network/byfn.sh#L504
dave.enyeart (Fri, 22 Mar 2019 01:03:17 GMT):
Is that something you want to change before initial release?
guoger (Fri, 22 Mar 2019 01:34:02 GMT):
just to summarize the discussion in slack a bit: we are solving two problems here:
- if leader is evicted, transfer leadership then halt
- if follower is evicted, at least leader should delay reconfiguring communication so that follower can have time to pick up conf change.
I agree we can defer #1 to v2.0 to avoid "invasive" change (it's not invasive in my opinion but i do empathize the paranoia).
but i think we should just add a delay, which is simpler than configuring on next block. basically what we would do is:
- if leader is evicted, leader halts *after* `ElectionTimeout` (it does *not* accept new tx). followers reconfigure communication immediately.
- if follower is evicted, leader reconfigures comm *after* `ElectionTimeout` (async). remaining followers reconfigure communication immediately, evicted follower halts immediately
i see two problems of reconfiguring on next block:
- we are relying on evicted leader to coordinate consensus
- there may not be next block soon in idle network
wdyt @yacovm @jyellick
guoger (Fri, 22 Mar 2019 01:34:02 GMT):
just to summarize the discussion in slack a bit: we are solving two problems here:
- if leader is evicted, transfer leadership then halt
- if follower is evicted, at least leader should delay reconfiguring communication so that follower can have time to pick up conf change.
I agree we can defer #1 to v2.0 to avoid "invasive" change (it's not invasive in my opinion but i do empathize the paranoia).
but i think we should just add a delay, which is simpler than configuring on next block. the minimal change i could think of is:
- if leader is evicted, leader halts *after* `ElectionTimeout` (it does *not* accept new tx). followers reconfigure communication immediately.
- if follower is evicted, no change needed
i see two problems of reconfiguring on next block:
- we are relying on evicted leader to coordinate consensus
- there may not be next block soon in idle network
wdyt @yacovm @jyellick
guoger (Fri, 22 Mar 2019 01:34:02 GMT):
just to summarize the discussion in slack a bit: we are solving two problems here:
- if leader is evicted, transfer leadership then halt
- if follower is evicted, at least leader should delay reconfiguring communication so that follower can have time to pick up conf change.
I agree we can defer #1 to v2.0 to avoid "invasive" change (it's not invasive in my opinion but i do empathize the paranoia).
but i think we should just add a delay, which is simpler than configuring on next block. the minimal change i could think of is:
- if leader is evicted, leader halts *after* `ElectionTimeout` (it does *not* accept new tx). no change needed for followers (they still reconfigure immediately).
- if follower is evicted, no change needed
i see two problems of reconfiguring on next block:
- we are relying on evicted leader to coordinate consensus
- there may not be next block soon in idle network
wdyt @yacovm @jyellick
guoger (Fri, 22 Mar 2019 01:34:02 GMT):
just to summarize the discussion in slack a bit: we are solving two problems here:
- if leader is evicted, transfer leadership then halt
- if follower is evicted, at least leader should delay reconfiguring communication so that follower can have time to pick up conf change.
I agree we can defer #1 to v2.0 to avoid "invasive" change (it's not invasive in my opinion but i do empathize the paranoia).
but i think we should just add a delay, which is simpler than configuring on next block. the minimal change i could think of is:
- if leader is evicted, leader halts *after* `ElectionTimeout` (this is async, and leader does *not* accept new tx in the mean time). no change needed for followers (they still reconfigure immediately).
- if follower is evicted, no change needed
i see two problems of reconfiguring on next block:
- we are relying on evicted leader to coordinate consensus
- there may not be next block soon in idle network
wdyt @yacovm @jyellick
guoger (Fri, 22 Mar 2019 03:59:44 GMT):
@jyellick @yacovm this should look less scary: https://gerrit.hyperledger.org/r/c/30297
tock (Fri, 22 Mar 2019 08:22:13 GMT):
@dave.enyeart thanks for merging this. I will cherry pick that, sure. the raft2 file is from the previous commit when there were 2 additional raft nodes... now there are 4... I can change that, no problem,
yacovm (Fri, 22 Mar 2019 09:02:25 GMT):
@guoger the code looks fine, and indeed less scary, but - do you remember if we have an integration test that evicts the leader? ;)
guoger (Fri, 22 Mar 2019 09:03:18 GMT):
yes we do. but it's pretty hard to reproduce this problem with integration test actually
guoger (Fri, 22 Mar 2019 09:04:05 GMT):
on UT, we can filter the messages, and simulate network partition, etc. but not IT :( @yacovm
yacovm (Fri, 22 Mar 2019 09:04:25 GMT):
why it's hard? find the leader and evict him
guoger (Fri, 22 Mar 2019 09:04:38 GMT):
we do have IT
guoger (Fri, 22 Mar 2019 09:04:51 GMT):
i'm just saying that IT doesn't actually reproduce the problem
yacovm (Fri, 22 Mar 2019 09:04:59 GMT):
it sometimes does no?
guoger (Fri, 22 Mar 2019 09:05:24 GMT):
i've never seen one... have you?
yacovm (Fri, 22 Mar 2019 09:05:58 GMT):
plenty of time, that's why I made the evicted node be the leader + 1 mod n
guoger (Fri, 22 Mar 2019 09:06:47 GMT):
hmmm, have you by any chance collected log URL or logs?
yacovm (Fri, 22 Mar 2019 09:07:08 GMT):
https://github.com/hyperledger/fabric/blob/release-1.4/integration/e2e/etcdraft_reconfig_test.go#L681-L706
yacovm (Fri, 22 Mar 2019 09:07:50 GMT):
> hmmm, have you by any chance collected log URL or logs?
lol, no
guoger (Fri, 22 Mar 2019 09:08:03 GMT):
```
By("Removing the leader from both system channel and application channel")
nwo.RemoveConsenter(network, peer, network.Orderers[(evictedNode+1)%3], "systemchannel", serverCertBytes)
```
are you referring to this?
yacovm (Fri, 22 Mar 2019 09:08:45 GMT):
so the removed node is actually the leader - 1 mod n
yacovm (Fri, 22 Mar 2019 09:08:55 GMT):
not + 1, i was confused. but same idea
yacovm (Fri, 22 Mar 2019 09:09:18 GMT):
i am saying that i changed it because evicting the leader broke the test 1 out of 3 times
guoger (Fri, 22 Mar 2019 09:09:33 GMT):
ah, so the `By` is not actually accurate. I'll change it back to evict leader
yacovm (Fri, 22 Mar 2019 09:09:39 GMT):
in the past i picked always the first node arbitrary
yacovm (Fri, 22 Mar 2019 09:09:50 GMT):
> ah, so the `By` is not actually accurate. I'll change it back to evict leader
correct...
guoger (Fri, 22 Mar 2019 09:20:45 GMT):
@yacovm wait... the server cert you prepared is still leader
```
By("Waiting for them to elect a leader")
evictedNode := findLeader(ordererRunners) - 1
By("Removing the leader from system channel")
serverCertBytes, err := ioutil.ReadFile(filepath.Join(network.OrdererLocalTLSDir(network.Orderers[evictedNode]), "server.crt"))
Expect(err).To(Not(HaveOccurred()))
By("Removing the leader from both system channel and application channel")
nwo.RemoveConsenter(network, peer, network.Orderers[(evictedNode+1)%3], "systemchannel", serverCertBytes)
fmt.Fprintln(GinkgoWriter, "Ensuring the other orderers detect the eviction of the node on channel", "systemchannel")
Eventually(ordererRunners[(evictedNode+1)%3].Err(), network.EventuallyTimeout, time.Second).Should(gbytes.Say("Deactivated node"))
Eventually(ordererRunners[(evictedNode+2)%3].Err(), network.EventuallyTimeout, time.Second).Should(gbytes.Say("Deactivated node"))
```
seems that it still evicts leader, no?
guoger (Fri, 22 Mar 2019 09:20:45 GMT):
@yacovm wait... the server cert you prepared is still leader
```
By("Waiting for them to elect a leader")
evictedNode := findLeader(ordererRunners) - 1
By("Removing the leader from system channel")
serverCertBytes, err := ioutil.ReadFile(filepath.Join(network.OrdererLocalTLSDir(network.Orderers[evictedNode]), "server.crt"))
Expect(err).To(Not(HaveOccurred()))
By("Removing the leader from both system channel and application channel")
nwo.RemoveConsenter(network, peer, network.Orderers[(evictedNode+1)%3], "systemchannel", serverCertBytes)
fmt.Fprintln(GinkgoWriter, "Ensuring the other orderers detect the eviction of the node on channel", "systemchannel")
Eventually(ordererRunners[(evictedNode+1)%3].Err(), network.EventuallyTimeout, time.Second).Should(gbytes.Say("Deactivated node"))
Eventually(ordererRunners[(evictedNode+2)%3].Err(), network.EventuallyTimeout, time.Second).Should(gbytes.Say("Deactivated node"))
```
seems that it still evicts leader, no? @yacovm
yacovm (Fri, 22 Mar 2019 09:26:33 GMT):
no
yacovm (Fri, 22 Mar 2019 09:26:42 GMT):
it evicts `leader-1`
yacovm (Fri, 22 Mar 2019 09:26:49 GMT):
and sends the config update to `leader+1`
guoger (Fri, 22 Mar 2019 09:28:50 GMT):
`evictedNode := findLeader(ordererRunners) - 1` is the *index* of leader, and the cert is read from `network.OrdererLocalTLSDir(network.Orderers[evictedNode]`, which is also the leader
guoger (Fri, 22 Mar 2019 09:29:14 GMT):
and if you inspect the log of that test, it *does* evicts leader. @yacovm
guoger (Fri, 22 Mar 2019 09:29:14 GMT):
and if you inspect the log of that test, it *does* evict leader. @yacovm
guoger (Fri, 22 Mar 2019 09:29:56 GMT):
`findLeader(ordererRunners)` returns the raft node id, which is index+1
yacovm (Fri, 22 Mar 2019 09:30:22 GMT):
but the cert it removes is `leader -1` no?
guoger (Fri, 22 Mar 2019 09:31:53 GMT):
`nwo.RemoveConsenter` basically compare certs in the list and cert provided, and removes it. so it's really removing `filepath.Join(network.OrdererLocalTLSDir(network.Orderers[evictedNode]`, which is leader
guoger (Fri, 22 Mar 2019 09:31:53 GMT):
`nwo.RemoveConsenter` basically compares certs in the list and cert provided, and removes it. so it's really removing `filepath.Join(network.OrdererLocalTLSDir(network.Orderers[evictedNode]`, which is leader
yacovm (Fri, 22 Mar 2019 09:32:45 GMT):
but `evictedNode := findLeader(ordererRunners) - 1`
yacovm (Fri, 22 Mar 2019 09:32:57 GMT):
so it's not the leader.. what am i missing?
guoger (Fri, 22 Mar 2019 09:34:45 GMT):
`findLeader(ordererRunners)` returns the *raft node ID*, which starts from 1 (not 0)
yacovm (Fri, 22 Mar 2019 09:34:55 GMT):
oh... right
yacovm (Fri, 22 Mar 2019 09:35:42 GMT):
ok so it means it always works then, no?
yacovm (Fri, 22 Mar 2019 09:35:49 GMT):
in the tests
yacovm (Fri, 22 Mar 2019 09:35:54 GMT):
what is the scenario you had in mind?
guoger (Fri, 22 Mar 2019 09:38:04 GMT):
yep, it always worked, because the corner case we are solving here is pretty hard to reproduce:
> evicted leader halts too fast, and hasn't had chance to instruct followers to commit the raft config change (which removes leader)
guoger (Fri, 22 Mar 2019 09:38:49 GMT):
if, in this case, remaining nodes can still form a quorum, we are fine because they eventually figures this out.
guoger (Fri, 22 Mar 2019 09:40:22 GMT):
however, if we are going from 2 -> 1, or 3/4 -> 2/3 (where 3/4 denotes 3 active nodes out of 4 in total), then we have a problem in this case
guoger (Fri, 22 Mar 2019 09:42:31 GMT):
so the leader should transfer the leader ship before halt (the "scary" CR). But we could halt with delay for the sake of simplicity (current CR) @yacovm
yacovm (Fri, 22 Mar 2019 09:52:10 GMT):
by figuring out eventually you mean that even if a node is too slow and is behind, the leader is evicted and there will be a new leader election where the leader would update the slow node, right? @guoger
yacovm (Fri, 22 Mar 2019 09:52:54 GMT):
and now when i think about what i said earlier - maybe it was the opposite actually, i think i made it explicit to remove the leader because sometimes removing a follower made it that the follower didn't get the update in time and the rest black-listed it.
yacovm (Fri, 22 Mar 2019 09:53:26 GMT):
but i still don't understand why we can't have an integration test that moves from 2 to 1
yacovm (Fri, 22 Mar 2019 09:53:30 GMT):
just to see that it works....
yacovm (Fri, 22 Mar 2019 09:53:32 GMT):
what do you thikn?
yacovm (Fri, 22 Mar 2019 09:53:32 GMT):
what do you think?
guoger (Fri, 22 Mar 2019 09:56:35 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=xQ5y6zHhwHgtAxG4m) @yacovm To be more accurate, even if slow nodes haven't applied conf change, at least 1 of 2 nodes has that conf change in storage, it just hasn't been instructed to commit it yet. So when leader halts, it ceases to send out heartbeats, and eventually a new leader gets elected, and continues the effort to commit the config change, and everything is ok
guoger (Fri, 22 Mar 2019 09:56:56 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=H4ng9GecwhZubFK9N) @yacovm I did and it works, again, because evicted leader never halts that fast
guoger (Fri, 22 Mar 2019 10:34:57 GMT):
@yacovm if a chain is halted, there's no `inactiveChain` created for it, correct?
guoger (Fri, 22 Mar 2019 10:47:54 GMT):
also, what's the default interval of checking if an inactive chain is part of channel?
guoger (Fri, 22 Mar 2019 10:47:54 GMT):
also, what's the default interval of checking if an inactive node is part of channel?
guoger (Fri, 22 Mar 2019 10:47:54 GMT):
~also, what's the default interval of checking if an inactive node is part of channel?~ got it, hardcode to 10 sec and exponentially increase up to 5 min
yacovm (Fri, 22 Mar 2019 10:54:52 GMT):
it's created at startup
jyellick (Fri, 22 Mar 2019 13:57:58 GMT):
@guoger @yacovm Trying to catch up here, but to summarize, it sounds like we're going to go with the simple 'delay' for now. Obviously we can test this in UT. If we added a 2->1 test in IT, would that cover us?
yacovm (Fri, 22 Mar 2019 15:39:41 GMT):
yes i believe so
guoger (Fri, 22 Mar 2019 17:48:55 GMT):
sure, I'll add an IT as well (or maybe just piggyback an existing one)
jyellick (Fri, 22 Mar 2019 18:05:09 GMT):
+1, let's please try to extend existing e2e tests instead of creating entirely new ones where possible.
jyellick (Fri, 22 Mar 2019 18:05:53 GMT):
Once release-1.4 is done, I'd actually like for us to look at consolidating as many of these Raft e2es into a single instance as is reasonable.
jyellick (Fri, 22 Mar 2019 18:05:53 GMT):
Once release-1.4 is done, I'd actually like for us to look at consolidating as many of these Raft e2es into as few networks as is reasonable.
jyellick (Fri, 22 Mar 2019 18:05:53 GMT):
Once v1.4.1 done, I'd actually like for us to look at consolidating as many of these Raft e2es into as few networks as is reasonable.
jyellick (Fri, 22 Mar 2019 18:05:53 GMT):
Once v1.4.1 is out, I'd actually like for us to look at consolidating as many of these Raft e2es into as few networks as is reasonable.
yacovm (Fri, 22 Mar 2019 18:35:09 GMT):
> +1, let's please try to extend existing e2e tests instead of creating entirely new ones where possible.
@jyellick this should be a test that runs 15 seconds tops...
yacovm (Fri, 22 Mar 2019 18:35:09 GMT):
> +1, let's please try to extend existing e2e tests instead of creating entirely new ones where possible.
@jyellick this should be a test that runs 15 seconds tops...
yacovm (Fri, 22 Mar 2019 18:35:09 GMT):
> +1, let's please try to extend existing e2e tests instead of creating entirely new ones where possible.
@jyellick this should be a test that runs 15 seconds tops...
yacovm (Fri, 22 Mar 2019 18:35:09 GMT):
> +1, let's please try to extend existing e2e tests instead of creating entirely new ones where possible.
this should be a test that runs 15 seconds tops... @jyellick
yacovm (Fri, 22 Mar 2019 18:37:16 GMT):
in the tests we usually test with 3 nodes... any test that we change would make us run with 2, no?
yacovm (Fri, 22 Mar 2019 18:37:33 GMT):
@guoger maybe you can extend https://github.com/hyperledger/fabric/blob/master/integration/e2e/etcdraft_reconfig_test.go#L683
yacovm (Fri, 22 Mar 2019 18:37:52 GMT):
and remove the 2nd node?
yacovm (Fri, 22 Mar 2019 18:37:56 GMT):
(the leader)
yacovm (Fri, 22 Mar 2019 18:38:10 GMT):
it should only add like a few seconds to total runtime
jyellick (Fri, 22 Mar 2019 19:24:00 GMT):
Exactly, @yacovm the heavyweight process of the integration tests is generating all the artifacts, starting all the processes, etc. Once the network is up, we should be able to reconfigure it to 2 nodes, then to 1 in a fairly trivial amount of time
yacovm (Fri, 22 Mar 2019 19:24:25 GMT):
but isn't the generation done once for the entire suite @jyellick ?
yacovm (Fri, 22 Mar 2019 19:24:45 GMT):
of the binaries
yacovm (Fri, 22 Mar 2019 19:24:55 GMT):
i think that generating the artifcats is not much time
yacovm (Fri, 22 Mar 2019 19:25:01 GMT):
most time is compilation of our binaries i think
jyellick (Fri, 22 Mar 2019 19:25:21 GMT):
The binary compilation happens once per suite
jyellick (Fri, 22 Mar 2019 19:25:21 GMT):
The binary compilation happens once for the whole suite
jyellick (Fri, 22 Mar 2019 19:26:07 GMT):
I am more thinking the generation of the config blocks, the generation of channel creation transactions, of crypto materials, starting the processes up, waiting for them to synchronize and begin accepting requests etc.
yacovm (Fri, 22 Mar 2019 19:26:21 GMT):
ok
yanli133 (Mon, 25 Mar 2019 06:55:16 GMT):
Has joined the channel.
tock (Mon, 25 Mar 2019 07:39:54 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=LMtnyLgYMHrFRxDpj) @dave.enyeart @dave.enyeart @jyellick @yacovm I cherry-picked the BYFN Raft sample to release-1.4, please review.
https://gerrit.hyperledger.org/r/#/c/30347/ & https://gerrit.hyperledger.org/r/#/c/30348/
tock (Mon, 25 Mar 2019 07:39:54 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=LMtnyLgYMHrFRxDpj) @dave.enyeart @jyellick @yacovm I cherry-picked the BYFN Raft sample to release-1.4, please review.
https://gerrit.hyperledger.org/r/#/c/30347/ & https://gerrit.hyperledger.org/r/#/c/30348/
guoger (Mon, 25 Mar 2019 07:44:57 GMT):
when an inactive chain is added to a channel, users might want it to participate in consensus more quickly, and might want to tune `ReplicationBackgroundRefreshInterval`. Should we expose this config as well? @yacovm
guoger (Mon, 25 Mar 2019 14:00:06 GMT):
@yacovm any reason why don't we want to expose this? and i've left a comment on Joe's configuration doc, thinking we should at least note this down, so that impatient user can restart node to force catch up
guoger (Mon, 25 Mar 2019 14:00:35 GMT):
also, i think we still have one thing to decide on: `Raft` vs `EtcdRaft`, which to use in our codebase/comments/logs/docs
guoger (Mon, 25 Mar 2019 14:00:35 GMT):
also, i think we still have one thing to decide on: `Raft` vs `EtcdRaft`, which to use in our codebase/comments/logs/docs, cc @jyellick @yacovm @C0rWin and other folks in this channel
yacovm (Mon, 25 Mar 2019 14:35:09 GMT):
I don't want to expose this because it's a tradeoff between how frequent you poll and waste resources vs. how fast you are added to the channel
yacovm (Mon, 25 Mar 2019 14:35:34 GMT):
you only add new OSNs very very rarely
yacovm (Mon, 25 Mar 2019 14:35:50 GMT):
and you anyway do this with coordination and out of band discussions
yacovm (Mon, 25 Mar 2019 14:35:55 GMT):
so 5 minutes is a small price to pay
guoger (Tue, 26 Mar 2019 04:05:12 GMT):
where do we guard against invalid raft options in channel creation tx? @jyellick
jyellick (Tue, 26 Mar 2019 04:20:54 GMT):
@guoger Channel creation is the same path as a channel config update, from a config processing point of view. However, I know some of the validation for the Raft options is done in the Raft package rather than in the mainline config path, so this bears some investigation.
guoger (Tue, 26 Mar 2019 04:21:30 GMT):
i'm investigating and i fear it's not validated :(
guoger (Tue, 26 Mar 2019 04:22:16 GMT):
if user submits a malformed channel creation tx, i.e. invalid consenter set, orderer panics while trying to create new chain
jyellick (Tue, 26 Mar 2019 04:23:24 GMT):
Obviously we cannot address a larger refactor for v1.4.1, but this is (yet another) good argument for why we should not be half-implementing configtx processing in the Raft package.
jyellick (Tue, 26 Mar 2019 04:23:24 GMT):
Obviously we cannot contain a larger refactor for v1.4.1, but this is (yet another) good argument for why we should not be half-implementing configtx processing in the Raft package.
jyellick (Tue, 26 Mar 2019 04:23:58 GMT):
Can you take it as a task to also validate the channel create tx consenter sets?
guoger (Tue, 26 Mar 2019 04:24:15 GMT):
working on it
guoger (Tue, 26 Mar 2019 04:24:45 GMT):
yeah.. we should've injected a validator to configtx processor
guoger (Tue, 26 Mar 2019 04:25:38 GMT):
sometimes, we may need some knowledge of current cluster status to validate configtx, i.e. don't allow node addition/removal if cluster quorum is lost, but this should be in another category
jyellick (Tue, 26 Mar 2019 04:26:12 GMT):
I still think this could easily be done in a plugin structure. We have the `ValidateNew` concept which takes as input the old and new configurations
guoger (Tue, 26 Mar 2019 04:26:56 GMT):
oh, so we feed `ValidateNew` with current status?
jyellick (Tue, 26 Mar 2019 04:27:46 GMT):
`ValidateNew` is an operation on the current config with the parameter being the new config, as things are structured today.
jyellick (Tue, 26 Mar 2019 04:28:15 GMT):
This is the plug point for instance where we disallow change of consensus type
guoger (Tue, 26 Mar 2019 04:29:48 GMT):
oh, what i meant was more of _"dynamic config"_, where *both* last configs *and* current runtime status are needed to validate new config
jyellick (Tue, 26 Mar 2019 04:30:17 GMT):
Oh, you need the status beyond what is in the current configuration?
guoger (Tue, 26 Mar 2019 04:30:59 GMT):
not in current code, but for example, we can disallow addition/removal/rotation if cluster is at minimal quorum
jyellick (Tue, 26 Mar 2019 04:31:39 GMT):
I'm sure we could figure out a scheme to address this, though probably not worth focusing too much on for the moment.
adarshsaraf123 (Tue, 26 Mar 2019 06:21:36 GMT):
@jyellick @guoger While Jay works on a quick fix for v1.4.1 on guarding against invalid raft options in channel creation tx, I can volunteer to look at the config refactoring to move validation of the Raft options out of the Raft package to the mainline config path.
adarshsaraf123 (Tue, 26 Mar 2019 06:21:36 GMT):
@jyellick @guoger While Jay works on a fix for v1.4.1 on guarding against invalid raft options in channel creation tx, I can volunteer to look at the config refactoring to move validation of the Raft options out of the Raft package to the mainline config path.
jyellick (Wed, 27 Mar 2019 14:50:52 GMT):
@guoger @C0rWin @yacovm @tock @adarshsaraf123 We are trying to (hopefully) close down a v1.4.1-rc here in the next 24 hours or so, so we need to make sure we get through these reviews. Below is the list of outstanding CRs. If I've missed something of yours, please speak up.
Jay:
https://gerrit.hyperledger.org/r/c/30297/ -- FAB-14764 halt with delay if it's removed as leader
https://gerrit.hyperledger.org/r/c/30270/ -- FAB-14745 more harsh snapshot integration test (Has 2+2)
https://gerrit.hyperledger.org/r/c/30299/ -- FAB-14766 improve logs text for serviceability
https://gerrit.hyperledger.org/r/c/30423/ -- FAB-14822 change func signature of CreateChannelFail
https://gerrit.hyperledger.org/r/c/30424/ -- FAB-14821 validate Raft config metadata
https://gerrit.hyperledger.org/r/c/30455/ -- FAB-14840 check Raft config for HeaderType_CONFIG
https://gerrit.hyperledger.org/r/c/30425/ -- FAB-14824 add integration test
Yacov:
https://gerrit.hyperledger.org/r/c/30372/ -- [FAB-14796] Warn about cert expiration - Part I
https://gerrit.hyperledger.org/r/c/30441/ -- [FAB-14802] Warn about cert expiration - Part II
Joe:
https://gerrit.hyperledger.org/r/c/30426/ -- [FAB-14404] Add Raft command to BYFN
jyellick (Wed, 27 Mar 2019 14:50:52 GMT):
@guoger @C0rWin @yacovm @tock @adarshsaraf123 @joe-alewine We are trying to (hopefully) close down a v1.4.1-rc here in the next 24 hours or so, so we need to make sure we get through these reviews. Below is the list of outstanding CRs. If I've missed something of yours, please speak up.
Jay:
https://gerrit.hyperledger.org/r/c/30297/ -- FAB-14764 halt with delay if it's removed as leader
https://gerrit.hyperledger.org/r/c/30270/ -- FAB-14745 more harsh snapshot integration test (Has 2+2)
https://gerrit.hyperledger.org/r/c/30299/ -- FAB-14766 improve logs text for serviceability
https://gerrit.hyperledger.org/r/c/30423/ -- FAB-14822 change func signature of CreateChannelFail
https://gerrit.hyperledger.org/r/c/30424/ -- FAB-14821 validate Raft config metadata
https://gerrit.hyperledger.org/r/c/30455/ -- FAB-14840 check Raft config for HeaderType_CONFIG
https://gerrit.hyperledger.org/r/c/30425/ -- FAB-14824 add integration test
Yacov:
https://gerrit.hyperledger.org/r/c/30372/ -- [FAB-14796] Warn about cert expiration - Part I
https://gerrit.hyperledger.org/r/c/30441/ -- [FAB-14802] Warn about cert expiration - Part II
Joe:
https://gerrit.hyperledger.org/r/c/30426/ -- [FAB-14404] Add Raft command to BYFN
jyellick (Wed, 27 Mar 2019 14:50:52 GMT):
@guoger @C0rWin @yacovm @tock @adarshsaraf123 @joe-alewine We are trying to (hopefully) close down a v1.4.1-rc here in the next 24 hours or so, so we need to make sure we get through these reviews. Below is the list of outstanding CRs. If I've missed something of yours, please speak up.
Jay:
https://gerrit.hyperledger.org/r/c/30297/ -- FAB-14764 halt with delay if it's removed as leader
https://gerrit.hyperledger.org/r/c/30270/ -- FAB-14745 more harsh snapshot integration test (Has 2+2)
https://gerrit.hyperledger.org/r/c/30299/ -- FAB-14766 improve logs text for serviceability
https://gerrit.hyperledger.org/r/c/30423/ -- FAB-14822 change func signature of CreateChannelFail
https://gerrit.hyperledger.org/r/c/30424/ -- FAB-14821 validate Raft config metadata
https://gerrit.hyperledger.org/r/c/30455/ -- FAB-14840 check Raft config for HeaderType_CONFIG
https://gerrit.hyperledger.org/r/c/30425/ -- FAB-14824 add integration test
Yacov:
https://gerrit.hyperledger.org/r/c/30372/ -- [FAB-14796] Warn about cert expiration - Part I
https://gerrit.hyperledger.org/r/c/30441/ -- [FAB-14802] Warn about cert expiration - Part II
Joe:
https://gerrit.hyperledger.org/r/c/30426/ -- [FAB-14404] Add Raft command to BYFN
https://gerrit.hyperledger.org/r/c/29449/ -- Orderer concept (needs to be cherry-picked to release-1.4)
https://gerrit.hyperledger.org/r/c/30010/ -- Configure Raft (needs to be cherry-picked to release-1.4)
https://gerrit.hyperledger.org/r/c/30242/ -- Stand up orderer node (implementation agnostic) (needs to be cherry-picked to release-1.4)
jyellick (Wed, 27 Mar 2019 14:50:52 GMT):
@guoger @C0rWin @yacovm @tock @adarshsaraf123 @joe-alewine We are trying to (hopefully) close down a v1.4.1-rc here in the next 24 hours or so, so we need to make sure we get through these reviews. Below is the list of outstanding CRs. If I've missed something of yours, please speak up.
Jay:
https://gerrit.hyperledger.org/r/c/30297/ -- FAB-14764 halt with delay if it's removed as leader
https://gerrit.hyperledger.org/r/c/30270/ -- FAB-14745 more harsh snapshot integration test (Has 2+2)
https://gerrit.hyperledger.org/r/c/30299/ -- FAB-14766 improve logs text for serviceability
https://gerrit.hyperledger.org/r/c/30423/ -- FAB-14822 change func signature of CreateChannelFail
https://gerrit.hyperledger.org/r/c/30424/ -- FAB-14821 validate Raft config metadata
https://gerrit.hyperledger.org/r/c/30455/ -- FAB-14840 check Raft config for HeaderType_CONFIG
https://gerrit.hyperledger.org/r/c/30425/ -- FAB-14824 add integration test
Yacov:
https://gerrit.hyperledger.org/r/c/30372/ -- [FAB-14796] Warn about cert expiration - Part I
https://gerrit.hyperledger.org/r/c/30441/ -- [FAB-14802] Warn about cert expiration - Part II
Joe:
https://gerrit.hyperledger.org/r/c/30426/ -- [FAB-14404] Add Raft command to BYFN
~https://gerrit.hyperledger.org/r/c/29449/ -- Orderer concept~ (needs to be cherry-picked to release-1.4)
https://gerrit.hyperledger.org/r/c/30010/ -- Configure Raft (needs to be cherry-picked to release-1.4)
https://gerrit.hyperledger.org/r/c/30242/ -- Stand up orderer node (implementation agnostic) (needs to be cherry-picked to release-1.4)
jyellick (Wed, 27 Mar 2019 14:50:52 GMT):
@guoger @C0rWin @yacovm @tock @adarshsaraf123 @joe-alewine We are trying to (hopefully) close down a v1.4.1-rc here in the next 24 hours or so, so we need to make sure we get through these reviews. Below is the list of outstanding CRs. If I've missed something of yours, please speak up.
Jay:
https://gerrit.hyperledger.org/r/c/30297/ -- FAB-14764 halt with delay if it's removed as leader
https://gerrit.hyperledger.org/r/c/30270/ -- FAB-14745 more harsh snapshot integration test (Has 2+2)
https://gerrit.hyperledger.org/r/c/30299/ -- FAB-14766 improve logs text for serviceability
https://gerrit.hyperledger.org/r/c/30423/ -- FAB-14822 change func signature of CreateChannelFail
https://gerrit.hyperledger.org/r/c/30424/ -- FAB-14821 validate Raft config metadata
https://gerrit.hyperledger.org/r/c/30455/ -- FAB-14840 check Raft config for HeaderType_CONFIG
https://gerrit.hyperledger.org/r/c/30425/ -- FAB-14824 add integration test
Yacov:
https://gerrit.hyperledger.org/r/c/30372/ -- [FAB-14796] Warn about cert expiration - Part I
https://gerrit.hyperledger.org/r/c/30441/ -- [FAB-14802] Warn about cert expiration - Part II
Joe:
https://gerrit.hyperledger.org/r/c/30426/ -- [FAB-14404] Add Raft command to BYFN
~https://gerrit.hyperledger.org/r/c/29449/ -- Orderer concept~ (needs to be cherry-picked to release-1.4)
~https://gerrit.hyperledger.org/r/c/30010/ -- Configure Raft~ (needs to be cherry-picked to release-1.4)
https://gerrit.hyperledger.org/r/c/30242/ -- Stand up orderer node (implementation agnostic) (needs to be cherry-picked to release-1.4)
jyellick (Wed, 27 Mar 2019 14:50:52 GMT):
@guoger @C0rWin @yacovm @tock @adarshsaraf123 @joe-alewine We are trying to (hopefully) close down a v1.4.1-rc here in the next 24 hours or so, so we need to make sure we get through these reviews. Below is the list of outstanding CRs. If I've missed something of yours, please speak up.
Jay:
~https://gerrit.hyperledger.org/r/c/30297/ -- FAB-14764 halt with delay if it's removed as leader~
~https://gerrit.hyperledger.org/r/c/30270/ -- FAB-14745 more harsh snapshot integration test (Has 2+2)~
~https://gerrit.hyperledger.org/r/c/30299/ -- FAB-14766 improve logs text for serviceability~
https://gerrit.hyperledger.org/r/c/30423/ -- FAB-14822 change func signature of CreateChannelFail
https://gerrit.hyperledger.org/r/c/30424/ -- FAB-14821 validate Raft config metadata
https://gerrit.hyperledger.org/r/c/30455/ -- FAB-14840 check Raft config for HeaderType_CONFIG
https://gerrit.hyperledger.org/r/c/30425/ -- FAB-14824 add integration test
Yacov:
https://gerrit.hyperledger.org/r/c/30372/ -- [FAB-14796] Warn about cert expiration - Part I
https://gerrit.hyperledger.org/r/c/30441/ -- [FAB-14802] Warn about cert expiration - Part II
Joe:
https://gerrit.hyperledger.org/r/c/30426/ -- [FAB-14404] Add Raft command to BYFN
~https://gerrit.hyperledger.org/r/c/29449/ -- Orderer concept~ (needs to be cherry-picked to release-1.4)
~https://gerrit.hyperledger.org/r/c/30010/ -- Configure Raft~ (needs to be cherry-picked to release-1.4)
https://gerrit.hyperledger.org/r/c/30242/ -- Stand up orderer node (implementation agnostic) (needs to be cherry-picked to release-1.4)
jyellick (Wed, 27 Mar 2019 14:50:52 GMT):
@guoger @C0rWin @yacovm @tock @adarshsaraf123 @joe-alewine We are trying to (hopefully) close down a v1.4.1-rc here in the next 24 hours or so, so we need to make sure we get through these reviews. Below is the list of outstanding CRs. If I've missed something of yours, please speak up.
Jay:
~https://gerrit.hyperledger.org/r/c/30297/ -- FAB-14764 halt with delay if it's removed as leader~
~https://gerrit.hyperledger.org/r/c/30270/ -- FAB-14745 more harsh snapshot integration test (Has 2+2)~
~https://gerrit.hyperledger.org/r/c/30299/ -- FAB-14766 improve logs text for serviceability~
https://gerrit.hyperledger.org/r/c/30423/ -- FAB-14822 change func signature of CreateChannelFail
https://gerrit.hyperledger.org/r/c/30424/ -- FAB-14821 validate Raft config metadata
https://gerrit.hyperledger.org/r/c/30455/ -- FAB-14840 check Raft config for HeaderType_CONFIG
https://gerrit.hyperledger.org/r/c/30425/ -- FAB-14824 add integration test
Yacov:
~https://gerrit.hyperledger.org/r/c/30372/ -- [FAB-14796] Warn about cert expiration - Part I~
~https://gerrit.hyperledger.org/r/c/30441/ -- [FAB-14802] Warn about cert expiration - Part II~
Joe:
https://gerrit.hyperledger.org/r/c/30426/ -- [FAB-14404] Add Raft command to BYFN
~https://gerrit.hyperledger.org/r/c/29449/ -- Orderer concept~ (needs to be cherry-picked to release-1.4)
~https://gerrit.hyperledger.org/r/c/30010/ -- Configure Raft~ (needs to be cherry-picked to release-1.4)
https://gerrit.hyperledger.org/r/c/30242/ -- Stand up orderer node (implementation agnostic) (needs to be cherry-picked to release-1.4)
jyellick (Wed, 27 Mar 2019 14:50:52 GMT):
@guoger @C0rWin @yacovm @tock @adarshsaraf123 @joe-alewine We are trying to (hopefully) close down a v1.4.1-rc here in the next 24 hours or so, so we need to make sure we get through these reviews. Below is the list of outstanding CRs. If I've missed something of yours, please speak up.
Jay:
~https://gerrit.hyperledger.org/r/c/30297/ -- FAB-14764 halt with delay if it's removed as leader~
~https://gerrit.hyperledger.org/r/c/30270/ -- FAB-14745 more harsh snapshot integration test (Has 2+2)~
~https://gerrit.hyperledger.org/r/c/30299/ -- FAB-14766 improve logs text for serviceability~
~https://gerrit.hyperledger.org/r/c/30423/ -- FAB-14822 change func signature of CreateChannelFail~
~https://gerrit.hyperledger.org/r/c/30424/ -- FAB-14821 validate Raft config metadata~
~https://gerrit.hyperledger.org/r/c/30455/ -- FAB-14840 check Raft config for HeaderType_CONFIG~
https://gerrit.hyperledger.org/r/c/30425/ -- FAB-14824 add integration test
Yacov:
~https://gerrit.hyperledger.org/r/c/30372/ -- [FAB-14796] Warn about cert expiration - Part I~
~https://gerrit.hyperledger.org/r/c/30441/ -- [FAB-14802] Warn about cert expiration - Part II~
Joe:
https://gerrit.hyperledger.org/r/c/30426/ -- [FAB-14404] Add Raft command to BYFN
~https://gerrit.hyperledger.org/r/c/29449/ -- Orderer concept~ (needs to be cherry-picked to release-1.4)
~https://gerrit.hyperledger.org/r/c/30010/ -- Configure Raft~ (needs to be cherry-picked to release-1.4)
https://gerrit.hyperledger.org/r/c/30242/ -- Stand up orderer node (implementation agnostic) (needs to be cherry-picked to release-1.4)
jyellick (Wed, 27 Mar 2019 14:50:52 GMT):
@guoger @C0rWin @yacovm @tock @adarshsaraf123 @joe-alewine We are trying to (hopefully) close down a v1.4.1-rc here in the next 24 hours or so, so we need to make sure we get through these reviews. Below is the list of outstanding CRs. If I've missed something of yours, please speak up.
Jay:
~https://gerrit.hyperledger.org/r/c/30297/ -- FAB-14764 halt with delay if it's removed as leader~
~https://gerrit.hyperledger.org/r/c/30270/ -- FAB-14745 more harsh snapshot integration test (Has 2+2)~
~https://gerrit.hyperledger.org/r/c/30299/ -- FAB-14766 improve logs text for serviceability~
~https://gerrit.hyperledger.org/r/c/30423/ -- FAB-14822 change func signature of CreateChannelFail~
~https://gerrit.hyperledger.org/r/c/30424/ -- FAB-14821 validate Raft config metadata~
~https://gerrit.hyperledger.org/r/c/30455/ -- FAB-14840 check Raft config for HeaderType_CONFIG~
https://gerrit.hyperledger.org/r/c/30425/ -- FAB-14824 add integration test
Yacov:
~https://gerrit.hyperledger.org/r/c/30372/ -- [FAB-14796] Warn about cert expiration - Part I~
~https://gerrit.hyperledger.org/r/c/30441/ -- [FAB-14802] Warn about cert expiration - Part II~
https://gerrit.hyperledger.org/r/c/30486/
Joe:
https://gerrit.hyperledger.org/r/c/30426/ -- [FAB-14404] Add Raft command to BYFN
~https://gerrit.hyperledger.org/r/c/29449/ -- Orderer concept~ (needs to be cherry-picked to release-1.4)
~https://gerrit.hyperledger.org/r/c/30010/ -- Configure Raft~ (needs to be cherry-picked to release-1.4)
https://gerrit.hyperledger.org/r/c/30242/ -- Stand up orderer node (implementation agnostic) (needs to be cherry-picked to release-1.4)
jyellick (Wed, 27 Mar 2019 14:50:52 GMT):
@guoger @C0rWin @yacovm @tock @adarshsaraf123 @joe-alewine We are trying to (hopefully) close down a v1.4.1-rc here in the next 24 hours or so, so we need to make sure we get through these reviews. Below is the list of outstanding CRs. If I've missed something of yours, please speak up.
Jay:
~https://gerrit.hyperledger.org/r/c/30297/ -- FAB-14764 halt with delay if it's removed as leader~
~https://gerrit.hyperledger.org/r/c/30270/ -- FAB-14745 more harsh snapshot integration test (Has 2+2)~
~https://gerrit.hyperledger.org/r/c/30299/ -- FAB-14766 improve logs text for serviceability~
~https://gerrit.hyperledger.org/r/c/30423/ -- FAB-14822 change func signature of CreateChannelFail~
~https://gerrit.hyperledger.org/r/c/30424/ -- FAB-14821 validate Raft config metadata~
~https://gerrit.hyperledger.org/r/c/30455/ -- FAB-14840 check Raft config for HeaderType_CONFIG~
https://gerrit.hyperledger.org/r/c/30425/ -- FAB-14824 add integration test
Yacov:
~https://gerrit.hyperledger.org/r/c/30372/ -- [FAB-14796] Warn about cert expiration - Part I~
~https://gerrit.hyperledger.org/r/c/30441/ -- [FAB-14802] Warn about cert expiration - Part II~
https://gerrit.hyperledger.org/r/c/30486/ -- [FAB-14852] Validate TLS certs are x509 encoded
Joe:
https://gerrit.hyperledger.org/r/c/30426/ -- [FAB-14404] Add Raft command to BYFN
~https://gerrit.hyperledger.org/r/c/29449/ -- Orderer concept~ (needs to be cherry-picked to release-1.4)
~https://gerrit.hyperledger.org/r/c/30010/ -- Configure Raft~ (needs to be cherry-picked to release-1.4)
https://gerrit.hyperledger.org/r/c/30242/ -- Stand up orderer node (implementation agnostic) (needs to be cherry-picked to release-1.4)
guoger (Wed, 27 Mar 2019 15:37:15 GMT):
mine is all captured
guoger (Wed, 27 Mar 2019 15:37:15 GMT):
mine is all captured in this list
guoger (Thu, 28 Mar 2019 09:14:01 GMT):
maybe orderer name should explicitly start from 1 in byfn? otherwise we have:
```
orderer.example.com
orderer2.example.com
orderer3.example.com
...
```
which is not super elegant IMO. (we could rename orderer to orderer1 in solo to avoid complicating scripts). Anyway this is just a nit. @tock
yacovm (Thu, 28 Mar 2019 13:06:04 GMT):
@jyellick I added https://gerrit.hyperledger.org/r/#/c/30486/ to the list
tock (Thu, 28 Mar 2019 16:46:23 GMT):
@guoger it does not start from 1 because solo and kafka use that name `orderer.example.com`. In an effort to keep the changes to a minimum and not change how solo and kafka behave, elegance was sacrificed.
guoger (Fri, 29 Mar 2019 01:34:43 GMT):
@tock yeah, that's why i thought we could alter that to be `orderer1.example.com` for solo/kafka. but i guess not worth doing this time
guoger (Fri, 29 Mar 2019 12:41:52 GMT):
I have two CRs out there for review, should be fairly easy, one adds an integration test and another one just addresses code hygiene from previous review comments. @jyellick @C0rWin @yacovm maybe someone can take a look?
guoger (Fri, 29 Mar 2019 12:58:10 GMT):
But I guess it’s fine if they are not included into rc
ronenschafferibm (Tue, 02 Apr 2019 12:02:10 GMT):
Has joined the channel.
corpix (Tue, 02 Apr 2019 12:44:50 GMT):
Has joined the channel.
klkumar369 (Tue, 02 Apr 2019 18:40:25 GMT):
Has joined the channel.
guoger (Wed, 03 Apr 2019 08:41:54 GMT):
two cherry-picks https://gerrit.hyperledger.org/r/c/30515/2 and https://gerrit.hyperledger.org/r/c/30425/9 need another +2. They were merged on master already. @yacovm could you take a look?
guoger (Wed, 03 Apr 2019 08:41:54 GMT):
two cherry-picks https://gerrit.hyperledger.org/r/c/30515/2 and https://gerrit.hyperledger.org/r/c/30425/9 need another +2. They were merged on master already. @yacovm @jyellick could you take a look?
gen_el (Wed, 03 Apr 2019 11:24:37 GMT):
Has joined the channel.
tock (Thu, 04 Apr 2019 10:29:24 GMT):
@guoger @yacovm here is a fresh unit test flake in etcdraft: https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/11346/console is that something you are familiar with?
guoger (Thu, 04 Apr 2019 10:35:50 GMT):
@tock could you create a JIRA? i'll submit a fix for it. thx!
guoger (Thu, 04 Apr 2019 10:35:50 GMT):
@tock ~could you create a JIRA? i'll submit a fix for it. thx!~ I'm creating one
yacovm (Thu, 04 Apr 2019 10:43:56 GMT):
@tock , I don't write ginkgo unit tests, against my beliefs.
guoger (Thu, 04 Apr 2019 13:11:54 GMT):
If we add nodes w/o updating endpoints, a newly started node might got stuck at bootstrap trying to pull blocks, if all of existing endpoints are unreachable, right? i.e.
- O1, O2, O3 in a channel
- only O1 and O2 are started
- add and start O4, O5
- kill O1 and O2
- start O3 with latest config block
O3, in theory, knows O4 and O5 from config block, and together they should be able to form a quorum. However, O3 would stuck at trying to pull blocks from O1 and O2. Am I missing anything? @yacovm
yacovm (Thu, 04 Apr 2019 13:14:58 GMT):
no, but i don't see why that is a problem?
yacovm (Thu, 04 Apr 2019 13:15:03 GMT):
I specified clearly in the document
yacovm (Thu, 04 Apr 2019 13:15:10 GMT):
that you should add the endpoints to the config
yacovm (Thu, 04 Apr 2019 13:15:18 GMT):
if you fail to do that.... well, too bad
guoger (Thu, 04 Apr 2019 13:37:23 GMT):
Good to know it's documented, i overlooked that
yacovm (Thu, 04 Apr 2019 13:38:41 GMT):
i hope it is, unless someone removed it
guoger (Thu, 04 Apr 2019 13:39:18 GMT):
just went check and it is :)
guoger (Thu, 04 Apr 2019 13:39:18 GMT):
just went check and it is there :)
dave.enyeart (Thu, 04 Apr 2019 13:47:10 GMT):
I wanted to follow-up on the topic of future-proofing the Raft implementation. Sounds like we need an audit of the future-proofness before we do our initial release. That is must-do before release in my opinion.
dave.enyeart (Thu, 04 Apr 2019 13:47:10 GMT):
I wanted to follow-up on the topic of future-proofing the Raft implementation. Sounds like we need an audit of the future-proofness before we do our initial release. Not just for the Jira in discussion, but overall. That is must-do before release in my opinion.
guoger (Thu, 04 Apr 2019 13:48:12 GMT):
so the jira we talked about is https://jira.hyperledger.org/browse/FAB-14799, and we need to decide on whether we want to augment current `ConsensusRequest` to have an extra field `Metadata`(tentative name), so it can be extended in the future. What we have now is:
```
// ConsensusRequest is a consensus specific message sent to a cluster member.
message ConsensusRequest {
string channel = 1;
bytes payload = 2;
}
```
guoger (Thu, 04 Apr 2019 13:48:41 GMT):
and what we do is:
```
// Consensus passes the given ConsensusRequest message to the raft.Node instance
func (c *Chain) Consensus(req *orderer.ConsensusRequest, sender uint64) error {
if err := c.isRunning(); err != nil {
return err
}
stepMsg := &raftpb.Message{}
if err := proto.Unmarshal(req.Payload, stepMsg); err != nil {
return fmt.Errorf("failed to unmarshal StepRequest payload to Raft Message: %s", err)
}
if err := c.Node.Step(context.TODO(), *stepMsg); err != nil {
return fmt.Errorf("failed to process Raft Step message: %s", err)
}
return nil
}
```
guoger (Thu, 04 Apr 2019 13:51:36 GMT):
We do have a field`Context []byte` within `raftpb.Message`, where we could squeeze in some data, but that's hacky IMO
guoger (Thu, 04 Apr 2019 13:53:24 GMT):
I don't know how much performance impact it would have w/ extra unmarshalling tho, but I think disk IO is still the bottleneck
guoger (Thu, 04 Apr 2019 13:54:00 GMT):
even if we are not disseminating the metrics within this field, i feel it's still worth to add it, for extensibility.
guoger (Thu, 04 Apr 2019 13:55:17 GMT):
back to the metrics, IMO it's still useful to have `ActiveNodes` metrics on *both* follower and leader. And this can be done by a simply pingpong service (no info needs to be piggybacked to HB), which is orthogonal to Raft
guoger (Thu, 04 Apr 2019 13:57:38 GMT):
Because currently, there's no way to tell whether my node is partitioned away from quorum based on metrics. We only have `IsLeader`, which is 0 (false) if it's actually follower, or partitioned (constantly starting campaign)
guoger (Thu, 04 Apr 2019 13:58:10 GMT):
(obviously one could try to submit a tx and see it fails with "no Raft leader"
guoger (Thu, 04 Apr 2019 13:59:25 GMT):
(we used to have `LeaderID` in initial metrics CR, but it was altered because it was thought to be less useful if ID cannot be related to Orderer canonical name)
dave.enyeart (Thu, 04 Apr 2019 14:08:13 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=zLYWryRfjkZX6Dr7f) Question - do we need to include a version number if the Raft messages to help with future-proofing?
dave.enyeart (Thu, 04 Apr 2019 14:08:13 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=zLYWryRfjkZX6Dr7f) Question - do we need to include a version number in the Raft messages to help with future-proofing?
dave.enyeart (Thu, 04 Apr 2019 14:08:13 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=zLYWryRfjkZX6Dr7f) Question - do we need to include a version number in the Raft messages to help with future-proofing and rolling upgrades?
yacovm (Thu, 04 Apr 2019 14:12:57 GMT):
No... if the etcd implementation of Raft supports mixed versions then it should do the versioning on its own. If it doesn't there is nothing we can do anyway
yacovm (Thu, 04 Apr 2019 14:13:12 GMT):
we don't put anything in the raft messages apart from the raft message itself
adarshsaraf123 (Thu, 04 Apr 2019 15:30:03 GMT):
@guoger wrt adding an opaque `Metadata` field to the `ConsensusRequest` message, isn't it unnecessary given that protobuf supports addition of fields such that the resulting proto would be backward-compatible? We can always add new fields (or even removing existing fields with slightly more effort) without having to worry about marshalling/unmarshalling errors.
guoger (Sun, 07 Apr 2019 10:45:23 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=ucfFDLpCyNcNkdF2S) @adarshsaraf123 i think that's correct! the question remains whether we want to tackle it within 1.4.1 or later.
guoger (Wed, 10 Apr 2019 03:35:49 GMT):
appreciate a round of review on:
https://gerrit.hyperledger.org/r/c/30621/6 enforce ElectionTimeout to be 10x greater than HeartbeatTimeout
https://gerrit.hyperledger.org/r/c/30622/7 increase interval of proactive campaign
(they are currently on master branch, and I think they should be included in 1.4.1 release as well. I'll cherry-pick once approved)
Fix flaky Raft UT:
https://gerrit.hyperledger.org/r/c/30722/ master
https://gerrit.hyperledger.org/r/c/30724/ release-1.4
Fix flaky Raft IT:
https://gerrit.hyperledger.org/r/c/30724/ already merged in master and need another +2 to get into 1.4
cc @jyellick @C0rWin @yacovm
guoger (Wed, 10 Apr 2019 03:35:49 GMT):
appreciate a round of review on:
https://gerrit.hyperledger.org/r/c/30621/6 enforce ElectionTimeout to be 10x greater than HeartbeatTimeout
https://gerrit.hyperledger.org/r/c/30622/7 increase interval of proactive campaign
(they are currently on master branch, and I think they should be included in 1.4.1 release as well. I'll cherry-pick once approved)
Fix flaky Raft UT:
https://gerrit.hyperledger.org/r/c/30722/ master
https://gerrit.hyperledger.org/r/c/30724/ release-1.4
Fix flaky Raft IT:
https://gerrit.hyperledger.org/r/c/30669/ already merged in master and need another +2 to get into 1.4
cc @jyellick @C0rWin @yacovm
jyellick (Wed, 10 Apr 2019 03:59:49 GMT):
@guoger I was looking at 30722, it's still not entirely clearly to me how this resolves flakiness (though I don't doubt that it does), could you explain a bit more what the flake was?
jyellick (Wed, 10 Apr 2019 03:59:49 GMT):
@guoger I was looking at 30722, it's still not entirely clearly to me how this resolves flakiness (though I don't doubt that it does), could you explain a bit more what the flake was and how this change addresses it?
jyellick (Wed, 10 Apr 2019 04:00:54 GMT):
(And the IT link you posted is a dup of the UT flake, probably unintentional?)
guoger (Wed, 10 Apr 2019 04:01:43 GMT):
just updated the link. and let me type the explanation up
guoger (Wed, 10 Apr 2019 04:05:40 GMT):
this is how communication works: let's say we remove A from
guoger (Wed, 10 Apr 2019 04:05:40 GMT):
this is how communication works: let's say we remove A from
guoger (Wed, 10 Apr 2019 04:06:49 GMT):
the CR changes mock configurator to reflect communication more accurately
jyellick (Wed, 10 Apr 2019 04:15:02 GMT):
So to summarize, the 'real' comm layer does not use bidirectional links, so, reconfiguration communication to evict a node will only terminate connections from the evicted node, but not connections to that evicted node?
jyellick (Wed, 10 Apr 2019 04:15:02 GMT):
So to summarize, the 'real' comm layer does not use bidirectional links, so, reconfiguring communication to evict a node will only terminate connections from the evicted node, but not connections to that evicted node?
guoger (Wed, 10 Apr 2019 04:16:44 GMT):
That's correct from caller point of view (Raft). Although I'm not too sure whether it actually terminates the underlying connection (other channels might still be using it)
guoger (Wed, 10 Apr 2019 04:18:51 GMT):
and just to elaborate a bit. We once had a CR to introduce delay before reconfiguring comm when a node finds itself being evicted. So we don't depend on this comm behavior. However that CR was thought to be scary to be shipped with 1.4.1
jyellick (Wed, 10 Apr 2019 04:19:09 GMT):
Right, I do recall that one
guoger (Wed, 10 Apr 2019 04:23:20 GMT):
As for https://gerrit.hyperledger.org/r/c/30621/ (election timeout >= 10 * heartbeat), it's optional (nice to have). We could simply document this in ops guide
yacovm (Wed, 10 Apr 2019 07:28:56 GMT):
@guoger @jyellick you also terminate streams *to* the evicted node....
guoger (Wed, 10 Apr 2019 07:33:45 GMT):
@yacovm i don't think so... `remotePeers` in chain.go does not return differently if node itself is removed
yacovm (Wed, 10 Apr 2019 08:32:27 GMT):
when you reconfigure and remove a node's ID - you can't send anymore to that ID
yacovm (Wed, 10 Apr 2019 08:32:34 GMT):
@guoger
yacovm (Wed, 10 Apr 2019 08:33:06 GMT):
when the reconfiguration takes place - all streams of evicted IDs are closed
yacovm (Wed, 10 Apr 2019 08:34:55 GMT):
https://github.com/hyperledger/fabric/blob/master/orderer/common/cluster/comm_test.go#L541
guoger (Wed, 10 Apr 2019 08:38:56 GMT):
yes, but i was talking about the comm configuration on the exact node that's being evicted
guoger (Wed, 10 Apr 2019 08:39:46 GMT):
from that node's point of view, it's basically no-op when reconfigure, right? @yacovm
yacovm (Wed, 10 Apr 2019 08:48:23 GMT):
yes since you never pass your own node's info to the comm layer
yacovm (Wed, 10 Apr 2019 08:48:28 GMT):
so if it is removed, it cannot tell
guoger (Wed, 10 Apr 2019 08:49:15 GMT):
exactly
guoger (Wed, 10 Apr 2019 08:49:23 GMT):
thx for confirming
guoger (Thu, 11 Apr 2019 03:13:48 GMT):
as i'm looking at our code w.r.t future proofing, i don't see any obvious limitation. Although i'm a little uneasy about `StepResponse`, since it's not currently being used. IIUC, it can only be of type `SubmitResponse` for now, and it is not actually used. Would appreciate another pair of eyes to see if fields defined in it are appropriate to be used in the future. (It looks good to me though) @jyellick
```
// StepResponse is a message received from a cluster member.
message StepResponse {
oneof payload {
SubmitResponse submit_res = 1;
}
}
```
```
// SubmitResponse returns a success
// or failure status to the sender.
message SubmitResponse {
string channel = 1;
// Status code, which may be used to programatically respond to success/failure.
common.Status status = 2;
// Info string which may contain additional information about the returned status.
string info = 3;
}
```
jyellick (Thu, 11 Apr 2019 03:28:12 GMT):
Thanks for looking at that @guoger I've also taken a look and things look appropriately extensible to me
guoger (Fri, 12 Apr 2019 07:14:46 GMT):
should we be more explicit about `!TLSEnabled && isClusterType`? current panic is stack is not very friendly:
```
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xacb884]
goroutine 1 [running]:
github.com/hyperledger/fabric/core/comm.(*GRPCServer).SetClientRootCAs(0xc00017c000, 0xc000498240, 0x4, 0x6, 0x0, 0x0)
/opt/gopath/src/github.com/hyperledger/fabric/core/comm/server.go:326 +0x1b4
github.com/hyperledger/fabric/orderer/common/server.updateTrustedRoots(0xc000430c00, 0x11563a0, 0xc000132700, 0xc00000e320, 0x1, 0x1)
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/main.go:752 +0x159a
github.com/hyperledger/fabric/orderer/common/server.Start.func1(0xc000132700)
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/main.go:151 +0xf4
github.com/hyperledger/fabric/common/channelconfig.(*BundleSource).Update(0xc0009c0210, 0xc000132700)
/opt/gopath/src/github.com/hyperledger/fabric/common/channelconfig/bundlesource.go:46 +0x7c
github.com/hyperledger/fabric/common/channelconfig.NewBundleSource(0xc000132700, 0xc00000e328, 0x1, 0x1, 0xc00035f020)
/opt/gopath/src/github.com/hyperledger/fabric/common/channelconfig/bundlesource.go:38 +0x72
github.com/hyperledger/fabric/orderer/common/multichannel.(*Registrar).newLedgerResources(0xc000498480, 0xc000350b90, 0xc000350b90)
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/multichannel/registrar.go:354 +0x379
github.com/hyperledger/fabric/orderer/common/multichannel.(*Registrar).Initialize(0xc000498480, 0xc00032d230)
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/multichannel/registrar.go:158 +0x242
github.com/hyperledger/fabric/orderer/common/server.initializeMultichannelRegistrar(0xc00003fe00, 0xc000430300, 0xc00042c1d0, 0x0, 0xc0004982d0, 0x1b25180, 0xc00042c320, 0x2, 0x2, 0xc00042c330, ...)
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/main.go:585 +0x315
github.com/hyperledger/fabric/orderer/common/server.Start(0x1013e09, 0x5, 0xc000540900)
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/main.go:158 +0x7a3
github.com/hyperledger/fabric/orderer/common/server.Main()
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/main.go:91 +0x1ce
main.main()
/opt/gopath/src/github.com/hyperledger/fabric/orderer/main.go:15 +0x20
```
https://jira.hyperledger.org/browse/FAB-15157
@yacovm
guoger (Fri, 12 Apr 2019 07:14:46 GMT):
should we be more explicit about `!TLSEnabled && isClusterType`? current panic is stack is not very friendly:
```
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xacb884]
goroutine 1 [running]:
github.com/hyperledger/fabric/core/comm.(*GRPCServer).SetClientRootCAs(0xc00017c000, 0xc000498240, 0x4, 0x6, 0x0, 0x0)
/opt/gopath/src/github.com/hyperledger/fabric/core/comm/server.go:326 +0x1b4
github.com/hyperledger/fabric/orderer/common/server.updateTrustedRoots(0xc000430c00, 0x11563a0, 0xc000132700, 0xc00000e320, 0x1, 0x1)
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/main.go:752 +0x159a
github.com/hyperledger/fabric/orderer/common/server.Start.func1(0xc000132700)
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/main.go:151 +0xf4
github.com/hyperledger/fabric/common/channelconfig.(*BundleSource).Update(0xc0009c0210, 0xc000132700)
/opt/gopath/src/github.com/hyperledger/fabric/common/channelconfig/bundlesource.go:46 +0x7c
github.com/hyperledger/fabric/common/channelconfig.NewBundleSource(0xc000132700, 0xc00000e328, 0x1, 0x1, 0xc00035f020)
/opt/gopath/src/github.com/hyperledger/fabric/common/channelconfig/bundlesource.go:38 +0x72
github.com/hyperledger/fabric/orderer/common/multichannel.(*Registrar).newLedgerResources(0xc000498480, 0xc000350b90, 0xc000350b90)
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/multichannel/registrar.go:354 +0x379
github.com/hyperledger/fabric/orderer/common/multichannel.(*Registrar).Initialize(0xc000498480, 0xc00032d230)
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/multichannel/registrar.go:158 +0x242
github.com/hyperledger/fabric/orderer/common/server.initializeMultichannelRegistrar(0xc00003fe00, 0xc000430300, 0xc00042c1d0, 0x0, 0xc0004982d0, 0x1b25180, 0xc00042c320, 0x2, 0x2, 0xc00042c330, ...)
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/main.go:585 +0x315
github.com/hyperledger/fabric/orderer/common/server.Start(0x1013e09, 0x5, 0xc000540900)
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/main.go:158 +0x7a3
github.com/hyperledger/fabric/orderer/common/server.Main()
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/main.go:91 +0x1ce
main.main()
/opt/gopath/src/github.com/hyperledger/fabric/orderer/main.go:15 +0x20
```
https://jira.hyperledger.org/browse/FAB-15157
@yacovm
guoger (Fri, 12 Apr 2019 07:14:46 GMT):
should we be more explicit about `!TLSEnabled && isClusterType`? current panic is stack is not very friendly:
```
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xacb884]
goroutine 1 [running]:
github.com/hyperledger/fabric/core/comm.(*GRPCServer).SetClientRootCAs(0xc00017c000, 0xc000498240, 0x4, 0x6, 0x0, 0x0)
/opt/gopath/src/github.com/hyperledger/fabric/core/comm/server.go:326 +0x1b4
github.com/hyperledger/fabric/orderer/common/server.updateTrustedRoots(0xc000430c00, 0x11563a0, 0xc000132700, 0xc00000e320, 0x1, 0x1)
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/main.go:752 +0x159a
github.com/hyperledger/fabric/orderer/common/server.Start.func1(0xc000132700)
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/main.go:151 +0xf4
github.com/hyperledger/fabric/common/channelconfig.(*BundleSource).Update(0xc0009c0210, 0xc000132700)
/opt/gopath/src/github.com/hyperledger/fabric/common/channelconfig/bundlesource.go:46 +0x7c
github.com/hyperledger/fabric/common/channelconfig.NewBundleSource(0xc000132700, 0xc00000e328, 0x1, 0x1, 0xc00035f020)
/opt/gopath/src/github.com/hyperledger/fabric/common/channelconfig/bundlesource.go:38 +0x72
github.com/hyperledger/fabric/orderer/common/multichannel.(*Registrar).newLedgerResources(0xc000498480, 0xc000350b90, 0xc000350b90)
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/multichannel/registrar.go:354 +0x379
github.com/hyperledger/fabric/orderer/common/multichannel.(*Registrar).Initialize(0xc000498480, 0xc00032d230)
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/multichannel/registrar.go:158 +0x242
github.com/hyperledger/fabric/orderer/common/server.initializeMultichannelRegistrar(0xc00003fe00, 0xc000430300, 0xc00042c1d0, 0x0, 0xc0004982d0, 0x1b25180, 0xc00042c320, 0x2, 0x2, 0xc00042c330, ...)
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/main.go:585 +0x315
github.com/hyperledger/fabric/orderer/common/server.Start(0x1013e09, 0x5, 0xc000540900)
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/main.go:158 +0x7a3
github.com/hyperledger/fabric/orderer/common/server.Main()
/opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/main.go:91 +0x1ce
main.main()
/opt/gopath/src/github.com/hyperledger/fabric/orderer/main.go:15 +0x20
```
https://jira.hyperledger.org/browse/FAB-15157
guoger (Fri, 12 Apr 2019 07:15:25 GMT):
@yacovm ^^
yacovm (Fri, 12 Apr 2019 10:13:13 GMT):
yeah yeah.... i know
yacovm (Fri, 12 Apr 2019 10:13:21 GMT):
I will take care of it :)
yacovm (Fri, 12 Apr 2019 11:50:21 GMT):
https://gerrit.hyperledger.org/r/#/c/30918/
jyellick (Fri, 12 Apr 2019 13:33:27 GMT):
https://github.com/hyperledger/fabric/blob/6c005dd53448d0a3f70a816b79749e5d9ea433ae/orderer/consensus/etcdraft/consenter.go#L264-L267
^ Why does this exist? We should never be calling viper outside of the main config parsing package, and I'm unsure why we would have introduce this here?
guoger (Fri, 12 Apr 2019 13:59:15 GMT):
@jyellick we have a field `Consensus` in `orderer.yaml`, which is consensus-specific. That code snippet is to decode the opaque data. It's calling a handy viper util, which doesn't actually require knowledge of viper instance from main.
guoger (Fri, 12 Apr 2019 13:59:15 GMT):
@jyellick we have a field `Consensus` in `orderer.yaml`, which is consensus-specific. That code snippet is to decode this opaque data. It's calling a handy viper util, which doesn't actually require knowledge of viper instance from main.
adarshsaraf123 (Mon, 15 Apr 2019 11:48:57 GMT):
I will unfortunately be missing today's scrum too due to another conflict. I have been working (at a really slow pace) on the refactoring of the config validation. The work is on track and I only have to add tests to the implementations which should be done latest by day after tomorrow.
adarshsaraf123 (Mon, 15 Apr 2019 11:53:21 GMT):
The WIPs are available at this stack: https://gerrit.hyperledger.org/r/#/c/30811/
guoger (Mon, 15 Apr 2019 13:46:08 GMT):
i want to revive two threads of conversation here:
1) if we can agree on the [approach proposed](https://jira.hyperledger.org/browse/FAB-14799?focusedCommentId=58872&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-58872) in the JIRA, I could start adding `ActiveNodes` metrics
2) I still think heavy load integration test is valuable and should be upstreamed. It performs certain operations - add node/kill node/restart node, etc - under heavy traffic (more than 1 blocks in flight). could someone reach out to Matt to see whether we want this differently? Also, i've written in-proc orderer clients as part of that work, which can be reused as well, if we want to move raft integration test to its own pkg to be self-contained, similar to gossip.
thoughts? @jyellick @yacovm
yacovm (Mon, 15 Apr 2019 14:23:03 GMT):
we should def. move the Raft integration tests to its own package
yacovm (Mon, 15 Apr 2019 14:23:07 GMT):
there is no question in that
yacovm (Mon, 15 Apr 2019 14:23:34 GMT):
as for the active nodes - I said my opinion in the JIRA....
guoger (Mon, 15 Apr 2019 14:32:09 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=959RZ8rao4djHoAkj) @yacovm how about the integration test with heavy traffic?
yacovm (Mon, 15 Apr 2019 15:16:37 GMT):
i think heavy ITs have a very good potential for flakes no? @guoger
jyellick (Mon, 15 Apr 2019 16:24:09 GMT):
We definitely should move 90% of the etcdraft integration tests to their own folder. We should ensure that there are no peer processes brought up, and for many of them, I expect we could operate purely on the system channel. We should then be able to interact with the orderers directly through the Broadcast/Deliver APIs, and, we can probably re-use the same network for many of the tests.
guoger (Tue, 16 Apr 2019 02:33:47 GMT):
ok, i'll take on it
kostas (Tue, 16 Apr 2019 03:40:50 GMT):
FWIW, I'm also of the opinion that ITs that do heavy traffic testing should not be part of the test suite. We mostly want to focus on the green path and make sure that things click together w/o issues.
kostas (Tue, 16 Apr 2019 03:41:44 GMT):
Someone might suggest that we still add them under a feature flag, but I think the chances of them not being actively maintained when this happens are non-significant.
guoger (Tue, 16 Apr 2019 04:02:30 GMT):
I guess my point is, we need to do this, _either at IT level, or lower_ (We do have UT that does reconfig while blocks are inflight, but that's just UT for chain.go), because IMO this *is* the green path - people reconfig channel while system is serving requests - and PTE/OTE doesn't really produce enough pressure on orderers: we barely have inflight blocks, and the chance of config *and* inflight blocks happen at the same time is even smaller.
Also, i've been using this test as gatekeeper for a while during development, and it makes lazy assertions, which fail much less often than other ITs.
At any rate, we don't need to prioritize this (i could just rebase it once in a while). Thanks for chiming in @kostas , glad to see you're surviving Cal :P
guoger (Tue, 16 Apr 2019 04:05:42 GMT):
btw, seeing this UT failure while running `orderer/common/cluster` pkg:
```
--- FAIL: TestSend/Send_step_succeed (0.06s)
yacovm (Tue, 16 Apr 2019 05:56:35 GMT):
Damn
yacovm (Tue, 16 Apr 2019 07:27:23 GMT):
ok i can reproduce that in once in 10,000 tests
guoger (Tue, 16 Apr 2019 07:28:41 GMT):
i guess i was "lucky" this morning and hit this within 100 runs
C0rWin (Tue, 16 Apr 2019 07:30:32 GMT):
it either this or your computer 100 slower than Yacov's
C0rWin (Tue, 16 Apr 2019 07:30:32 GMT):
it either this or your computer 100 times slower than Yacov's
yacovm (Tue, 16 Apr 2019 07:31:00 GMT):
i didn't run with data race enabled
C0rWin (Tue, 16 Apr 2019 07:31:23 GMT):
oh, now it's fair
yacovm (Tue, 16 Apr 2019 07:32:43 GMT):
oh found the problem
yacovm (Tue, 16 Apr 2019 07:33:15 GMT):
```
sent <- struct{}{}
atomic.AddUint32(&sendCalls, 1)
```
needed to be
```
atomic.AddUint32(&sendCalls, 1)
sent <- struct{}{}
```
yacovm (Tue, 16 Apr 2019 07:36:12 GMT):
ok 100,000 runs and didn't fail
guoger (Tue, 16 Apr 2019 07:37:04 GMT):
how did you complete 100,000 runs within 1 minute?
yacovm (Tue, 16 Apr 2019 07:38:34 GMT):
151 seconds
yacovm (Tue, 16 Apr 2019 07:38:54 GMT):
Clipboard - April 16, 2019 10:38 AM
yacovm (Tue, 16 Apr 2019 07:40:03 GMT):
https://jira.hyperledger.org/browse/FAB-15201
guoger (Tue, 16 Apr 2019 07:42:57 GMT):
@yacovm what's `// expected 2, actual 1` supposed to mean?
guoger (Tue, 16 Apr 2019 07:43:13 GMT):
two calls result in one stream?
yacovm (Tue, 16 Apr 2019 08:11:18 GMT):
the test has 2 calls
yacovm (Tue, 16 Apr 2019 08:11:26 GMT):
2 sends
yacovm (Tue, 16 Apr 2019 08:11:39 GMT):
it ensures only 1 stream is actually created
yacovm (Tue, 16 Apr 2019 08:11:44 GMT):
despite you calling send twice
tock (Tue, 16 Apr 2019 14:39:01 GMT):
```{"level":"info","msg":"Periodic check is stopping.","channel":"multi-node-channel","node":3}
\u2022\u2022\u2022\u2022\u2022\u20222019-04-16 14:20:34.965 UTC [test] ReceiverByChain -> WARN 020 Chain notraftchain is of type *multichannel.ChainSupport and not etcdraft.Chain
\u20222019-04-16 14:20:34.968 UTC [test] ReceiverByChain -> PANI 021 Programming error - Chain badChainObject is nil although it exists in the mapping
\u20222019-04-16 14:20:34.970 UTC [test] HandleChain -> INFO 022 EvictionSuspicion not set, defaulting to 10m0s
2019-04-16 14:20:34.971 UTC [test] createOrReadWAL -> INFO 023 No WAL data found, creating new WAL at path '/tmp/snap-597125359/wal-' channel= node=1
2019-04-16 14:20:34.979 UTC [test] Start -> INFO 024 Starting Raft node channel= node=1
2019-04-16 14:20:34.980 UTC [test] start -> INFO 025 Starting raft node as part of a new channel channel= node=1
2019-04-16 14:20:34.980 UTC [test] becomeFollower -> INFO 026 1 became follower at term 0 channel= node=1
2019-04-16 14:20:34.981 UTC [test] newRaft -> INFO 027 newRaft 1 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0] channel= node=1
2019-04-16 14:20:34.981 UTC [test] becomeFollower -> INFO 028 1 became follower at term 1 channel= node=1
\u20222019-04-16 14:20:34.984 UTC [test] detectSelfID -> WARN 029 Could not find cert.orderer0.org0 among [cert.orderer1.org1]
\u20222019-04-16 14:20:34.985 UTC [test] run -> INFO 02a This node is picked to start campaign channel= node=1
2019-04-16 14:20:34.987 UTC [test] apply -> INFO 02b Applied config change to add node 1, current nodes in channel: [1] channel= node=1
\u20222019-04-16 14:20:34.990 UTC [test] HandleChain -> INFO 02c EvictionSuspicion not set, defaulting to 10m0s
\u2022
Summarizing 1 Failure:
[Fail] Chain 3-node Raft cluster when reconfiguring raft cluster reconfiguration when two type B config are sent back-to-back [It] discards the second
/home/tock/go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain_test.go:2074
Ran 101 of 101 Specs in 21.322 seconds
FAIL! -- 100 Passed | 1 Failed | 0 Pending | 0 Skipped
--- FAIL: TestEtcdraft (21.33s)
2019-04-16 14:20:35.486 UTC [test] Step -> INFO 02d 1 is starting a new election at term 1 channel= node=1
2019-04-16 14:20:35.486 UTC [test] becomePreCandidate -> INFO 02e 1 became pre-candidate at term 1 channel= node=1
2019-04-16 14:20:35.486 UTC [test] poll -> INFO 02f 1 received MsgPreVoteResp from 1 at term 1 channel= node=1
2019-04-16 14:20:35.487 UTC [test] becomeCandidate -> INFO 030 1 became candidate at term 2 channel= node=1
2019-04-16 14:20:35.487 UTC [test] poll -> INFO 031 1 received MsgVoteResp from 1 at term 2 channel= node=1
2019-04-16 14:20:35.487 UTC [test] becomeLeader -> INFO 032 1 became leader at term 2 channel= node=1
2019-04-16 14:20:35.487 UTC [test] run -> INFO 033 raft.node: 1 elected leader 1 at term 2 channel= node=1
2019-04-16 14:20:35.488 UTC [test] run -> INFO 034 Leader 1 is present, quit campaign channel= node=1
2019-04-16 14:20:35.488 UTC [test] serveRequest -> INFO 035 Raft leader changed: 0 -> 1 channel= node=1
2019-04-16 14:20:35.488 UTC [test] serveRequest -> INFO 036 Start accepting requests as Raft leader at block [1] channel= node=1
2019-04-16 14:20:38.406 UTC [test] Stop -> INFO 037 Periodic check is stopping.
FAIL
coverage: 85.6% of statements
FAIL github.com/hyperledger/fabric/orderer/consensus/etcdraft 30.818s
? github.com/hyperledger/fabric/orderer/consensus/etcdraft/mocks [no test files]
ok github.com/hyperledger/fabric/orderer/consensus/inactive (cached) coverage: 100.0% of statements
```
Unit test flake on etcdraft... is it familiar? @guoger
guoger (Tue, 16 Apr 2019 15:21:07 GMT):
@tock thx for reporting. do you have the link? log pasted here does not reflect actual failure
yacovm (Tue, 16 Apr 2019 19:26:40 GMT):
and I got:
```
21:22:30 Summarizing 1 Failure:
21:22:30
21:22:30 [Fail] Chain 3-node Raft cluster when reconfiguring raft cluster reconfiguration [It] stop cluster quorum and continue reconfiguration after the restart
21:22:30 /w/workspace/fabric-verify-unit-tests-x86_64/gopath/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain_test.go:3905
21:22:30
21:22:30 Ran 101 of 101 Specs in 37.217 seconds
21:22:30 FAIL! -- 100 Passed | 1 Failed | 0 Pending | 0 Skipped
21:22:30 --- FAIL: TestEtcdraft (37.22s)
```
in https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/11552/console
yacovm (Tue, 16 Apr 2019 19:26:56 GMT):
@guoger
guoger (Wed, 17 Apr 2019 02:40:24 GMT):
thx, will post a fix
tock (Wed, 17 Apr 2019 06:16:20 GMT):
@guoger that was in my local `build all`, not CI, so I do not have a link
guoger (Wed, 17 Apr 2019 06:25:54 GMT):
@tock sure np, i have a fix and about to push. thx
guoger (Wed, 17 Apr 2019 06:25:54 GMT):
@tock ~sure np, i have a fix and about to push. thx~ actually, the fix i was talking about was for the bug reported by yacov...
guoger (Wed, 17 Apr 2019 06:25:54 GMT):
@tock ~sure np, i have a fix and about to push. thx~ actually, the fix i was talking about was for the bug reported by yacov... i'll try to reproduce the one you posted. thx
guoger (Thu, 18 Apr 2019 13:09:43 GMT):
Folks, I’m gonna miss the scrum today, sorry. I just have couple CR for flakes for review and working on moving raft IT out
jyellick (Sun, 21 Apr 2019 18:04:16 GMT):
@guoger @tock @yacovm @C0rWin @adarshsaraf123 You likely all already saw my cancellation notice for the Raft scrum, but I wanted to share that my wife ended up going into labor a couple weeks ahead of schedule, and our new baby has arrived. Everyone involved is happy and healthy, but, my availability, especially over the next few weeks, will be very limited.
tock (Sun, 21 Apr 2019 18:10:33 GMT):
Congratulations!
yacovm (Sun, 21 Apr 2019 18:15:01 GMT):
@jyellick congrats :)
adarshsaraf123 (Sun, 21 Apr 2019 18:39:26 GMT):
@jyellick Congratulations! :smiley:
C0rWin (Sun, 21 Apr 2019 19:56:56 GMT):
@jyellick congrats!
jyellick (Sun, 21 Apr 2019 21:27:19 GMT):
Thanks all!
guoger (Mon, 22 Apr 2019 01:11:16 GMT):
congrats!
tock (Mon, 29 Apr 2019 14:55:41 GMT):
@guoger @yacovm Is this a flake or is it something that I did? https://jenkins.hyperledger.org/job/fabric-verify-integration-tests-x86_64/8405/console
yacovm (Mon, 29 Apr 2019 14:56:09 GMT):
```
text canceled" grpc.code=Unknown grpc.call_duration=31.803193ms
15:21:54 [e][OrdererOrg.orderer5] panic: failed to initialize operations subsystem: listen tcp 127.0.0.1:36034: bind: address already in use
15:21:54 [e][OrdererOrg.orderer5]
```
yacovm (Mon, 29 Apr 2019 14:56:12 GMT):
something you did
tock (Mon, 29 Apr 2019 15:00:44 GMT):
by selecting the last config over the bootstrap file?
tock (Mon, 29 Apr 2019 15:01:44 GMT):
```
opsSystem := newOperationsSystem(conf.Operations, conf.Metrics)
err := opsSystem.Start()
if err != nil {
logger.Panicf("failed to initialize operations subsystem: %s", err)
}
```
tock (Mon, 29 Apr 2019 15:01:44 GMT):
```
opsSystem := newOperationsSystem(conf.Operations, conf.Metrics)
err := opsSystem.Start()
if err != nil {
logger.Panicf("failed to initialize operations subsystem: %s", err)
}
```
tock (Mon, 29 Apr 2019 15:04:32 GMT):
how is that possible?
tock (Tue, 30 Apr 2019 10:21:28 GMT):
@yacovm @guoger Unit test flake in etcdraft: https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/11755/console
guoger (Tue, 30 Apr 2019 10:22:42 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=aWZo8CnJPqvbijnbp) @tock i'll look into it, thx!
tock (Wed, 08 May 2019 11:57:52 GMT):
anyone experiencing problems with gerrit?
mauricio (Sun, 12 May 2019 03:51:20 GMT):
Has left the channel.
tock (Tue, 14 May 2019 13:22:30 GMT):
@guoger Is this a well known flaky test? https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/12112/console
tock (Tue, 14 May 2019 14:02:55 GMT):
does anyone know if this is flaky?
16:46:56 FAIL github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/validator/statebasedval 32.301s
https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/12116/console
tock (Tue, 14 May 2019 14:02:55 GMT):
@sykesm does anyone know if this is flaky?
16:46:56 FAIL github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/validator/statebasedval 32.301s
https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/12116/console
guoger (Tue, 14 May 2019 15:38:02 GMT):
should be new... i'll put it to my tabs
guoger (Tue, 14 May 2019 15:58:48 GMT):
created FAB-15448
tock (Wed, 15 May 2019 06:11:06 GMT):
:thumbsup:
tock (Thu, 16 May 2019 08:37:17 GMT):
@jyellick @sykesm @yacovm @C0rWin Can you please take a look at these two?
https://gerrit.hyperledger.org/r/#/c/31335/
https://gerrit.hyperledger.org/r/#/c/31196/
tock (Thu, 16 May 2019 08:37:17 GMT):
@jyellick @sykesm @yacovm @C0rWin Can you please take a look at these two?
https://gerrit.hyperledger.org/r/#/c/31335/
https://gerrit.hyperledger.org/r/#/c/31196/
tock (Thu, 16 May 2019 08:37:17 GMT):
@jyellick @sykesm @yacovm @C0rWin Can you please take a look at these two CRs on migration?
https://gerrit.hyperledger.org/r/#/c/31335/
https://gerrit.hyperledger.org/r/#/c/31196/
C0rWin (Thu, 16 May 2019 08:38:32 GMT):
sure
shrivastava.amit (Thu, 16 May 2019 12:37:24 GMT):
Has joined the channel.
circlespainter (Sat, 18 May 2019 07:38:45 GMT):
Has joined the channel.
tock (Wed, 22 May 2019 08:37:14 GMT):
@yacovm @ronenschafferibm `11:11:28 FAIL github.com/hyperledger/fabric/orderer/common/cluster 11.284s` a flake in cluster?
https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/12274/console
tock (Wed, 22 May 2019 08:37:14 GMT):
@yacovm @ronenschafferibm `11:11:28 FAIL github.com/hyperledger/fabric/orderer/common/cluster 11.284s` a flake in cluster unit-test?
https://jenkins.hyperledger.org/job/fabric-verify-unit-tests-x86_64/12274/console
yacovm (Wed, 22 May 2019 09:15:24 GMT):
yes
yacovm (Wed, 22 May 2019 09:15:34 GMT):
looks like the test sometimes gets out of sync:
```
11:11:28 2019-05-22 08:10:22.715 UTC [test] pullBlocks -> ERRO 0c3 Received a bad block from 127.0.0.1:33318: response is of type
tock (Wed, 22 May 2019 13:37:38 GMT):
do you have a Jira for fixing it?
yacovm (Wed, 22 May 2019 15:48:43 GMT):
i had... i thought i fixed it
guoger (Mon, 27 May 2019 07:44:35 GMT):
@yacovm what do you think of removing `orderer/common/server/etcdraft_test.go`? it should be sufficiently covered by current IT (or, if anything is missing there, should be added)
yacovm (Mon, 27 May 2019 07:45:08 GMT):
why? it's a UT
guoger (Mon, 27 May 2019 07:45:29 GMT):
it's a UT that builds and starts OSN bin
yacovm (Mon, 27 May 2019 07:45:34 GMT):
i know that
yacovm (Mon, 27 May 2019 07:45:38 GMT):
but integration tests are super slow
yacovm (Mon, 27 May 2019 07:45:46 GMT):
and that test tests stuff that we don't test in IT
guoger (Mon, 27 May 2019 07:48:10 GMT):
don't we already have tests for sanity check of block?
yacovm (Mon, 27 May 2019 07:48:51 GMT):
huh?
guoger (Mon, 27 May 2019 07:50:35 GMT):
i see 3 things being tested there:
- invalid gensis block
- tls is not enabled
- raft OSN is lauchable
first two should already be covered by UT. as for the third, i think it's in the scope of IT
yacovm (Mon, 27 May 2019 07:51:52 GMT):
our UT testing capability in that package isn't good
yacovm (Mon, 27 May 2019 07:52:03 GMT):
so i think launching a binary and testing it can not hurt
yacovm (Mon, 27 May 2019 07:52:21 GMT):
as for the third - we're launching it with a separate listener. we don't do that in ITs
guoger (Mon, 27 May 2019 08:00:47 GMT):
you mean cluster listener and ab listener? I feel if we really want to test this, it should be done in IT, so we not only check it can launch, but also work properly in a multi-node set up. I'm allured to remove this test is because it's a slippery slope to add slower-than-UT-faster-than-IT tests into packages.
yacovm (Mon, 27 May 2019 08:02:39 GMT):
> it's a slippery slope to add slower-than-UT-faster-than-IT tests into packages.
right, that's exactly why it's there. It saves time overall because it's faster than ITs which run serially.
guoger (Mon, 27 May 2019 08:12:07 GMT):
sure... just for the sake of argument, it is not significantly faster than IT (once we don't launch chaincode or ordering specific tests), and tests a lot less than IT.
guoger (Mon, 27 May 2019 08:12:07 GMT):
sure... just for the sake of argument, it is not significantly faster than IT (once we don't launch chaincode in ordering specific tests), and tests a lot less than IT.
yacovm (Mon, 27 May 2019 08:12:52 GMT):
It is.
yacovm (Mon, 27 May 2019 08:13:12 GMT):
it runs in parallel to other tests
guoger (Mon, 27 May 2019 08:16:58 GMT):
well, if you are talking about time-to-finish in CI, it doesn't matter since we wait for IT anyway. If you are talking about local development, i don't think it increases the confidence... but not worth much debating i guess
yacovm (Mon, 27 May 2019 08:17:39 GMT):
That's exactly the problem. By converting it to IT you're only prolonging the total build time
guoger (Mon, 27 May 2019 08:17:56 GMT):
no i'm not saying coverting it to IT
yacovm (Mon, 27 May 2019 08:18:12 GMT):
well i don't want to remove it as it tests important stuff
guoger (Mon, 27 May 2019 08:18:43 GMT):
ok
yacovm (Mon, 27 May 2019 08:19:07 GMT):
let's hear more opinions... @jyellick what do you think?
yacovm (Mon, 27 May 2019 08:19:17 GMT):
should we remove the said UT ?
jyellick (Mon, 27 May 2019 19:17:31 GMT):
My take would be leave it in UT. If the IT tests become sufficiently fast and we cover all of the UT stuff in IT (which sounds like it is not currently the case), then we can reexamine. Nothing wrong with IT style tests in unit test paths if they make sense there.
tock (Mon, 03 Jun 2019 15:10:31 GMT):
@jyellick @sykesm @yacovm There are 5 CRs waiting for review, some are +2 by one reviewer:
https://gerrit.hyperledger.org/r/#/c/fabric/+/31196/ integration tests; +2 by Yacov, Matt, can you renew your +2? it was overridden by the last patch.
https://gerrit.hyperledger.org/r/c/fabric/+/31440 maintenance-filter; Jason, please take a look...
https://gerrit.hyperledger.org/r/c/fabric/+/31670 added a test case to integration to test the maintenance-filter
https://gerrit.hyperledger.org/r/c/fabric/+/31671 sizefilter bug fix, easy
These four are stacked. This one is unrelated, based on master:
https://gerrit.hyperledger.org/r/c/fabric/+/31672
Thanks!
dave.enyeart (Mon, 03 Jun 2019 19:56:13 GMT):
@jyellick @guoger @sykesm @tock With Jason resurfacing the daily Matt/Jason scrum will be the primary place for ordering work coordination. Please try to attend!
dave.enyeart (Mon, 03 Jun 2019 19:56:56 GMT):
Obviously the top focus for v1.4.2 is the kafka to raft migration, including system test @suryalanka and doc
suryalanka (Mon, 03 Jun 2019 19:56:56 GMT):
Has joined the channel.
dave.enyeart (Mon, 03 Jun 2019 19:57:33 GMT):
Also the fabric-orderer bugs need to be triaged, Jay could you help with these?
dave.enyeart (Mon, 03 Jun 2019 19:57:33 GMT):
Also the fabric-orderer bugs need to be triaged, Jay could you help with these (unless Jason peeks first)
dave.enyeart (Mon, 03 Jun 2019 19:57:35 GMT):
- FAB-15200: orderer discards the config message - Jay
- FAB-15558: Consenter for channel exiting - Jay
guoger (Tue, 04 Jun 2019 03:36:54 GMT):
FAB-15200: user would need to copy ledger data to new orderer to avoid bootstrapping from genesis. It is not a bug.
FAB-15558: waiting for more info to reproduce. Most likely it's a problem in Kafka (either bug or misconfiguration)
tock (Tue, 04 Jun 2019 06:06:46 GMT):
I'll start back-porting kafka-raft migration to 1.4.2 right away. The last two of the CRs are in review on master, so there is a decent amount of CRs merged already to start this task.
dave.enyeart (Tue, 04 Jun 2019 13:15:01 GMT):
Right, we need all the functional items in both master and release-1.4. For refactoring of tests and code, it's a case by case decision by the squad team on how much to keep the release branches in sync vs invest in master only.
yacovm (Tue, 04 Jun 2019 14:55:49 GMT):
@tock has only a single change set left to be merged to master and I +2d it, we need 1 more volunteer ;)
jyellick (Tue, 04 Jun 2019 20:03:52 GMT):
@tock What is the rationale behind not allowing config changes other than consensus type changes in maintenance mode? It seems generally useful that the network operator might want to for instance modify batch parameters before switching over consensus type, to add new certificate authorities, etc. Did you see some problem scenario?
guoger (Wed, 05 Jun 2019 03:16:08 GMT):
+1, I think this would be useful
guoger (Wed, 05 Jun 2019 04:31:08 GMT):
https://gerrit.hyperledger.org/r/c/fabric/+/31715 fixes an Raft IT flake, which happens fairly often after recent change. The change is simply (1 line). Appreciate if someone could take a look @jyellick @yacovm @C0rWin
tock (Wed, 05 Jun 2019 06:49:20 GMT):
I was thinking that all those changes could be done on-line either before or after migration, as is supported today. consensus-type migration is the only config change so far that requires maintenance mode, and my approach was restrictive: in order to maximize the probability of success, change one thing at a time.
However, if you guys think it is useful to permit all Orderer group changes in maintenance mode, that is an easy fix.
@yacovm what do you think?
tock (Wed, 05 Jun 2019 06:49:20 GMT):
I was thinking that all those changes could be done on-line either before or after migration, as is supported today. consensus-type migration is the only config change so far that requires maintenance mode, and my approach was restrictive: in order to maximize the probability of success, change one thing at a time.
However, if you guys think it is useful to permit all Orderer group changes in maintenance mode, that is an easy fix.
@yacovm what do you think?
tock (Wed, 05 Jun 2019 06:49:20 GMT):
I was thinking that all those changes could be done on-line either before or after migration, as is supported today. consensus-type migration is the only config change so far that requires maintenance mode, and my approach was restrictive: in order to maximize the probability of success, change one thing at a time.
However, if you guys think it is useful to permit all Orderer group changes in maintenance mode, that is easy to change.
@yacovm what do you think?
tock (Wed, 05 Jun 2019 06:49:20 GMT):
?
jyellick (Wed, 05 Jun 2019 14:33:34 GMT):
@adarshsaraf123 are you going to be able to finish up the raft config processing cleanup soon? If you can't get to it in the next week or so, I think we might need to hand it off to someone with cycles to complete it.
adarshsaraf123 (Wed, 05 Jun 2019 14:34:11 GMT):
Yes Jason. I am currently working on the reviews from Matt and Jay.
jyellick (Wed, 05 Jun 2019 14:34:33 GMT):
Ah great, thanks!
adarshsaraf123 (Wed, 05 Jun 2019 14:35:58 GMT):
So i was occupied till the end of May with a conference deadline and therefore I couldn't get to it so far.
jyellick (Wed, 05 Jun 2019 15:20:25 GMT):
No problem, I understand
tock (Wed, 05 Jun 2019 18:14:18 GMT):
The idea is that changing every other parameter is possible on-line, either before or after migration. Since establishing a new cluster is a sensitive operation, I was thinking of making it conservative and allowing one change at a time; first establish a raft cluster, then make changes.
However, if you guys think it is useful to allow other Orderer group config changes in maintenance mode, then the restriction can be easily removed. @yacovm what do you think?
jyellick (Wed, 05 Jun 2019 18:21:48 GMT):
I would even go so far as to allow arbitrary config changes. I would of doc that during consensus migration that we recommend one step at a time, but unless we have a concrete scenario where something destructive can happen, I'm not sure it's worth explicitly checking -- it is after all code we have to maintain and if we find a case in the future where maintenance mode for other config changes are useful, then we have to do yet another capability around the config processing.
jyellick (Wed, 05 Jun 2019 18:21:48 GMT):
I would even go so far as to allow arbitrary config changes. I would doc that during consensus migration that we recommend one step at a time, but unless we have a concrete scenario where something destructive can happen, I'm not sure it's worth explicitly checking -- it is after all code we have to maintain and if we find a case in the future where maintenance mode for other config changes are useful, then we have to do yet another capability around the config processing.
tock (Wed, 05 Jun 2019 18:30:53 GMT):
OK, sounds good. What I think should be kept is the following: 1) entry and exit to/from maintenance mode are not accompanied by a change; 2) ConsensusType.Type can only change in maintenance-mode. How does that sound?
jyellick (Wed, 05 Jun 2019 18:31:37 GMT):
Works for me, thanks Yoav!
tock (Thu, 06 Jun 2019 19:47:31 GMT):
I pushed a CR that implements this.
tock (Thu, 06 Jun 2019 19:47:31 GMT):
I pushed a CR that implements this. @jyellick
tock (Thu, 06 Jun 2019 19:47:34 GMT):
https://gerrit.hyperledger.org/r/c/fabric/+/31749
jyellick (Thu, 06 Jun 2019 19:50:01 GMT):
Great, I'll take a look
jyellick (Thu, 06 Jun 2019 19:50:14 GMT):
Great, I'll take a look
jyellick (Thu, 06 Jun 2019 20:08:07 GMT):
@tock https://gerrit.hyperledger.org/r/c/fabric/+/31750 is a simple cherry-pick of the code used to explicitly set capabilities that @sykesm was asking for in your series.
tock (Fri, 07 Jun 2019 05:18:43 GMT):
Looks good, I'll start using that in my integration test. V2_0 on master and V1_4_2 on release-v1.4. Thanks!
jyellick (Fri, 07 Jun 2019 18:45:36 GMT):
@tock Can you explain to me why this check is needed? https://github.com/hyperledger/fabric/blob/228387d00f18791b74f50fbb68f1bbd5c5730137/orderer/consensus/etcdraft/chain.go#L323 Could we not simply check to see if the consensus metadata field is nil in the block (as the block writer nils this field on migration), indicating that the chain must have been migrated?
jyellick (Fri, 07 Jun 2019 18:45:36 GMT):
@tock Can you explain to me why this check is needed? https://github.com/hyperledger/fabric/blob/228387d00f18791b74f50fbb68f1bbd5c5730137/orderer/consensus/etcdraft/chain.go#L323 Could we not simply check to see if the consensus metadata field is nil in the block (as the block writer nils this field on migration), indicating that the chain must have been migrated?
Particularly, if we can get away without having the `DetectConsensusMigration` method added to the consenter API, it would be nice.
yacovm (Fri, 07 Jun 2019 18:47:18 GMT):
what consensus metadata do you have when you have the genesis block?
jyellick (Fri, 07 Jun 2019 18:47:41 GMT):
Should be `nil` as well
yacovm (Fri, 07 Jun 2019 18:48:03 GMT):
well then how do you distinguish?
jyellick (Fri, 07 Jun 2019 18:48:18 GMT):
Block number? Also... it's fine not to? Both mean we need to initialize a new Raft WAL
yacovm (Fri, 07 Jun 2019 18:50:30 GMT):
oh you're saying that if our chain is non empty and we have a nil metadata then it means we have migrated ?
jyellick (Fri, 07 Jun 2019 18:50:35 GMT):
Right
jyellick (Fri, 07 Jun 2019 18:51:26 GMT):
Metadata should only ever be nil if 1) It's a new chain, 2) It's a migrated chain. And both are treated the same unless I'm missing something.
jyellick (Fri, 07 Jun 2019 18:52:51 GMT):
(And we can of course differentiate 1 and 2 by block number/height)
jyellick (Fri, 07 Jun 2019 18:52:51 GMT):
(And we can of course differentiate 1 and 2 by block number/height, though the only reason I might is for logging)
yacovm (Fri, 07 Jun 2019 18:53:36 GMT):
makes sense I think.... let's hear what @tock thinks
tock (Tue, 11 Jun 2019 06:51:25 GMT):
Checking that the block metadata is nil inside the chain.go seems like dark pattern that nobody but us will get, unless we disclose that secret... and it does not carry over to other consensus types. Hence the decision to implement it outside in method that is accessible to all `DetectConsensusMigration`. I think that is the right thing to do in the long term. Whether it is in the consenter API or soem other API that is a different issue, We can certainly move that (maybe when we dismantle the "Support" pattern?).
tock (Tue, 11 Jun 2019 06:51:25 GMT):
Checking that the block metadata is nil inside the chain.go seems like dark pattern that nobody but us will get, unless we disclose that secret... and it does not carry over to other consensus types. Hence the decision to implement it outside in a method that is accessible to all `DetectConsensusMigration`. I think that is the right thing to do in the long term. Whether it is in the consenter API or soem other API that is a different issue, We can certainly move that (maybe when we dismantle the "Support" pattern?).
tock (Tue, 11 Jun 2019 06:51:25 GMT):
Checking that the block metadata is nil inside the chain.go seems like dark pattern that nobody but us will get, unless we disclose that secret... and it does not carry over to other consensus types. Hence the decision to implement it outside in a method that is accessible to all `DetectConsensusMigration`. I think that is the right thing to do in the long term. Whether it is in the consenter API or some other API - that is a different issue, We can certainly move that (maybe when we dismantle the "Support" pattern?).
tock (Tue, 11 Jun 2019 06:53:42 GMT):
As to what is checked in `DetectConsensusMigration` , I strongly think that checking just metadata==nil is fragile... I rather be declarative and explicit about what is going on, I think although this may seem redundant now, it may prove easier to maintain and extend.
tock (Tue, 11 Jun 2019 06:53:42 GMT):
As to what is checked in `DetectConsensusMigration` , I strongly think that checking just metadata==nil is fragile... I rather be declarative and explicit about what is going on, I think although this may seem redundant now, it may prove easier to maintain and extend. That is why Detect is comparing the consensus-type values in to consecutive config blocks.
tock (Tue, 11 Jun 2019 06:53:42 GMT):
As to what is checked in `DetectConsensusMigration` , I strongly think that checking just metadata==nil is fragile... I rather be declarative and explicit about what is going on, I think although this may seem redundant now, it may prove easier to maintain and extend. That is why Detect is comparing the consensus-type values in two consecutive config blocks.
tock (Tue, 11 Jun 2019 06:56:43 GMT):
The way chain.go was written is mainly because I did not want to change the node.go API at that point in time. But that can also be refactored easily.
tock (Tue, 11 Jun 2019 06:56:43 GMT):
The way chain.go was written is mainly because I did not want to change the node.go start() at that point in time. But that can also be refactored easily.
jyellick (Tue, 11 Jun 2019 14:13:11 GMT):
> unless we disclose that secret... and it does not carry over to other consensus types.
We explicitly nil the consensus metadata when consensus type changes, how would other types be different?
jyellick (Tue, 11 Jun 2019 14:24:11 GMT):
But I'm not sure why this is such a dark pattern. Seems like we can explicitly state: "If the consensus implementation maintains state which must persist between reboots, then it should be stored in the consensus metadata of the block. If the consensus metadata is nil, then the consensus implementation must initialize this state and persist it in the next block. This typically occurs at channel creation time, or consensus type migration". That coalesces two distinct paths (bootstrap, and migration) into one code path, and removes a dependency from the API.
jyellick (Tue, 11 Jun 2019 14:24:11 GMT):
But I'm not sure why this is such a dark pattern. Seems like we can explicitly state: "If the consensus implementation maintains state which must persist between process restarts, then it should be stored in the consensus metadata of the block. If the consensus metadata is nil, then the consensus implementation must initialize this state and persist it in the next block. This typically occurs at channel creation time, or consensus type migration". That coalesces two distinct paths (bootstrap, and migration) into one code path, and removes a dependency from the API.
jyellick (Tue, 11 Jun 2019 14:24:11 GMT):
But I'm not sure why this is such a dark pattern. Seems like we can explicitly state: "If the consensus implementation maintains state which must persist between process restarts, then it should be stored in the consensus metadata of the block. If the consensus metadata is nil, then the consensus implementation must initialize this state and persist it in the next block. This typically occurs at channel creation time, or consensus type migration". That coalesces two distinct paths (bootstrap, and migration) into one code path, and removes a dependency from the API. It also seems quite natural and a standard pattern "If a thing is uninitialized, initialize it"
jyellick (Tue, 11 Jun 2019 14:28:05 GMT):
FWIW I also think having the more general ability to reset consenter state by nil-ing metadata would be useful. If a Raft cluster has permanently lost quorum, then simply build a ledger which is the union of all chains, zero the metadata, and start the cluster.
jyellick (Tue, 11 Jun 2019 14:28:05 GMT):
FWIW I also think having the more general ability to reset consenter state by nil-ing metadata would be useful. If a Raft cluster has permanently lost quorum, then simply build a ledger which is the union of all chains, nil the metadata, and start the cluster.
jyellick (Tue, 11 Jun 2019 14:28:29 GMT):
What do you think @tock ?
Swhit210 (Tue, 11 Jun 2019 17:06:18 GMT):
Has joined the channel.
tock (Wed, 12 Jun 2019 14:07:35 GMT):
I am not sure how resetting the metadata would help in this scenario... the orderer will get the last config block and there it will find the same set of nodes out of which there is no quorum... the metadata just provides indexes, no? In addition, the metadata is signed, so unless I am mistaken there is no easy way to change it "from the outside", no?
jyellick (Wed, 12 Jun 2019 14:13:51 GMT):
I was thinking of a scenario with a corrupted WAL
jyellick (Wed, 12 Jun 2019 14:15:01 GMT):
So, simply bootstrap the same set of Raft nodes, all with the same ledger info, all with nil-ed metadata, and they will restart the cluster successfully.
jyellick (Wed, 12 Jun 2019 14:16:28 GMT):
We could similarly do offline ledger modifications to add a config block with an updated consenter set. But we need some way to signal etcdraft that we are bootstrapping -- nil metadata would be an obvious mechanism.
jyellick (Wed, 12 Jun 2019 14:20:13 GMT):
Regardless of how useful nil metadata for bootstrapping is in other scenarios, let's return to the 'dark pattern' bit. Keying off nil-metadata would remove a hundred or so lines of code and test, plus a consenter interface dependency. What is the problem scenario you see? If there's no explicit problem (regardless of any other benefits), it seems like a win.
jyellick (Wed, 12 Jun 2019 14:20:23 GMT):
( @tock )
tock (Wed, 12 Jun 2019 18:48:17 GMT):
@jyellick I see that as a form of defensive programming. On one end of we insert a nil metadata only if: it is a consensus-type change, it is in maintenance, on two consecutive config blocks. On the other end, after restart, we check that if it is a nil, said conditions still hold. Why check? because there are so many ways things can go wrong, but only one way to be correct and consistent. Now, looking at the code, I see now that the checks I did are not good enough... they should probably panic when the metadata is nil and the above conditions do not hold; instead, they return false. That would probably make the node crash later on, but I don't know that for sure.
jyellick (Wed, 12 Jun 2019 18:50:20 GMT):
@tock Can you enumerate some situation where the metadata is nil that is not a) bootstrap, b) migration, or c) manual user intervention. Are any of these recoverable?
tock (Wed, 12 Jun 2019 18:54:08 GMT):
Well, here is one: after you and I have transitioned to another project, someone decides to insert nil metadata somewhere. Then, migration_test.go would blow up in his face and he would be informed that he needs to take care of these restrictions. I don't see how (c) is possible, as the metadata needs to be signed by an orderer...
jyellick (Wed, 12 Jun 2019 18:56:01 GMT):
I don't follow your example. How does someone decide to simply set nil metadata? If they need persistent state in the metadata, this would break them in all sorts of other ways?
jyellick (Wed, 12 Jun 2019 18:56:37 GMT):
Also, `migration_test.go` doesn't execute against the various consensus plugins anyway, so I'm not sure how this would be caught.
tock (Wed, 12 Jun 2019 19:00:44 GMT):
I meant programmatically, by implementing a code change to fabric...
jyellick (Wed, 12 Jun 2019 19:01:47 GMT):
Can you give a more concrete example? We have an API contract for consensus plugins which says they give us consensus metadata, and, on startup, we pass them the most recent copy of said metadata.
jyellick (Wed, 12 Jun 2019 19:01:47 GMT):
Can you give a more concrete example? We have an API contract for consensus plugins which says they give us consensus metadata for each block, and, on startup, we pass them the most recent copy of said metadata.
tock (Wed, 12 Jun 2019 19:06:50 GMT):
We nil the metadata whe we switch from one type to the other. what if someone changes blockwriter to nil the metadata on some other criteria? then, if he does not take care of the "Detect" check, his code will fail.
jyellick (Wed, 12 Jun 2019 19:07:40 GMT):
If someone changes blockwriter to nil the metadata... then consensus plugins will break, because it breaks the contract that we return the most recent metadata for that plugin on startup
tock (Wed, 12 Jun 2019 19:09:40 GMT):
well, maybe - I am not sure how it will break exactly, but it better break early, with a clear message, no?
tock (Wed, 12 Jun 2019 19:10:45 GMT):
kind of like "fail fast, fail safe"
jyellick (Wed, 12 Jun 2019 19:10:46 GMT):
It just doesn't seem like a particularly reasonable thing to defend against to me. If someone starts nil-ing data structures given to it for storage by a consensus plugin, what other result can they expect beyond failure? There are a multitude of ways that a bad programmer could corrupt the data given to blockwriter, but we don't defend against those.
jyellick (Wed, 12 Jun 2019 19:11:07 GMT):
What would the clear message be though?
tock (Wed, 12 Jun 2019 19:11:59 GMT):
"unexpected nil metadata, not during migration" ?
jyellick (Wed, 12 Jun 2019 19:12:37 GMT):
But some consensus plugins don't require metadata, and nil is perfectly acceptable (e.g. solo)
jyellick (Wed, 12 Jun 2019 19:13:05 GMT):
And solo doesn't care whether you're migrating from Kafka, or Raft, or anything. As there's no special init it needs to do on migration.
tock (Wed, 12 Jun 2019 19:16:33 GMT):
I mean, it is similar to when one component checks the validity of inputs given to it by a different component. In reality, even "good" programmers don't always know all the conditions the other component is expecting; but a robust system will reject those fast, when he breaks them.
tock (Wed, 12 Jun 2019 19:17:46 GMT):
we never migrate from anything to solo... I made sure of that ;-)
jyellick (Wed, 12 Jun 2019 19:18:11 GMT):
Which, if you saw my comment on your WIP CR I'd really like to change :slight_smile:
tock (Wed, 12 Jun 2019 19:20:53 GMT):
yep, i saw that and responded too... my approach is, if it is not tested, it is not working, so kafak->solo can be added with one line, but only with a corresponding integration test ;-)
jyellick (Wed, 12 Jun 2019 19:21:54 GMT):
I am still extremely unconvinced that this check is worth performing, but let me go ahead and proceed under the assumption that it's the right thing to do. I still don't think we're approaching it the right way. We've added a new function to the consenter plugin API, which we expect the consensus plugin to invoke during the chain startup, to see if there's extra work it needs to do. This seems like a case where a programmer very likely will forget to do this. Instead, it makes much more sense to me that we should simply detect the migration at startup, and pass in a field to the HandleChain function for the implementer to deal with.
jyellick (Wed, 12 Jun 2019 19:22:50 GMT):
> yep, i saw that and responded too... my approach is, if it is not tested, it is not working, so kafak->solo can be added with one line, but only with a corresponding integration test ;-)
We don't have to claim support for it.
jyellick (Wed, 12 Jun 2019 19:22:50 GMT):
> yep, i saw that and responded too... my approach is, if it is not tested, it is not working, so kafak->solo can be added with one line, but only with a corresponding integration test ;-)
We don't have to claim support for it. But we also don't need to maintain code which explicitly prevents it.
jyellick (Wed, 12 Jun 2019 19:23:45 GMT):
I didn't see an integration test which proves it can't be done either :slight_smile:
jyellick (Wed, 12 Jun 2019 19:23:45 GMT):
I didn't see an integration test which proves it can't be done either :slight_smile: (And let's be clear, we don't need to test red paths in integration tests, or even all paths)
tock (Wed, 12 Jun 2019 19:25:34 GMT):
ok,
tock (Wed, 12 Jun 2019 19:25:52 GMT):
```//=== Step 7: ===
By("7) Config update on system channel, change ConsensusType.Type to unsupported type, forbidden")
assertTransitionFailed(network, peer, orderer, syschannel,
"kafka", protosorderer.ConsensusType_STATE_MAINTENANCE,
"solo", nil, protosorderer.ConsensusType_STATE_MAINTENANCE)
//=== Step 8: ===
By("8) Config update on standard channel, change ConsensusType.Type to unsupported type, forbidden")
assertTransitionFailed(network, peer, orderer, channel1,
"kafka", protosorderer.ConsensusType_STATE_MAINTENANCE,
"hesse", nil, protosorderer.ConsensusType_STATE_MAINTENANCE)```
jyellick (Wed, 12 Jun 2019 19:26:55 GMT):
My mistake, I guess there is an integration test, but that's really tangential.
tock (Wed, 12 Jun 2019 19:27:14 GMT):
anyway, lets strive towards a solution...
tock (Wed, 12 Jun 2019 19:27:50 GMT):
I understand you concern. can you elaborate more about the alternative?
tock (Wed, 12 Jun 2019 19:27:50 GMT):
I understand your concern. can you elaborate more about the alternative?
jyellick (Wed, 12 Jun 2019 19:31:33 GMT):
Sure. Today, we have:
```// Consenter defines the backing ordering mechanism.
type Consenter interface {
// HandleChain should create and return a reference to a Chain for the given set of resources.
// It will only be invoked for a given chain once per process. In general, errors will be treated
// as irrecoverable and cause system shutdown. See the description of Chain for more details
// The second argument to HandleChain is a pointer to the metadata stored on the `ORDERER` slot of
// the last block committed to the ledger of this Chain. For a new chain, this metadata will be
// nil, as this field is not set on the genesis block
HandleChain(support ConsenterSupport, metadata *cb.Metadata) (Chain, error)
}
```
My proposed solution would be to simply do this:
```
// Consenter defines the backing ordering mechanism.
type Consenter interface {
// HandleChain should create and return a reference to a Chain for the given set of resources.
// It will only be invoked for a given chain once per process. In general, errors will be treated
// as irrecoverable and cause system shutdown. See the description of Chain for more details
// The second argument to HandleChain is a pointer to the metadata stored on the `ORDERER` slot of
// the last block committed to the ledger of this Chain. For a new chain, or one which is migrated,
// this metadata will be nil, as there is no prior metadata to report.
HandleChain(support ConsenterSupport, metadata *cb.Metadata) (Chain, error)
}
```
However, if we think we need to explicitly check whether a consensus migration occurred, I would do the following:
```
// Consenter defines the backing ordering mechanism.
type Consenter interface {
// HandleChain should create and return a reference to a Chain for the given set of resources.
// It will only be invoked for a given chain once per process. In general, errors will be treated
// as irrecoverable and cause system shutdown. See the description of Chain for more details
// The second argument to HandleChain is a pointer to the metadata stored on the `ORDERER` slot of
// the last block committed to the ledger of this Chain. For a new chain, this metadata will be
// nil, as this field is not set on the genesis block. For new chains, and for migrated chains, initialize
// will be set to true, indicating that the consenter should perform whatever one time initialization
// is required for its operation.
HandleChain(support ConsenterSupport, metadata *cb.Metadata, initialize bool) (Chain, error)
}
```
jyellick (Wed, 12 Jun 2019 19:33:28 GMT):
Then, the caller of `HandleChain` (chainsupport.go) would perform the detection checks and set the `initialize` var as appropriate.
tock (Wed, 12 Jun 2019 19:34:23 GMT):
ok, I understand. That sounds good.
tock (Wed, 12 Jun 2019 19:36:56 GMT):
Let me sleep on it (literally, it is getting late here ;-) ) and I'll take care of that first thing tomorrow morning.
jyellick (Wed, 12 Jun 2019 19:37:36 GMT):
Sounds good, thanks for staying up to discuss.
tock (Wed, 12 Jun 2019 19:39:04 GMT):
Sure, no problem.
ArtemFrantsiian (Fri, 14 Jun 2019 08:27:58 GMT):
Has joined the channel.
tock (Mon, 17 Jun 2019 12:23:02 GMT):
@jyellick These two CRs address the issues discussed above:
https://gerrit.hyperledger.org/r/c/fabric/+/31932
https://gerrit.hyperledger.org/r/c/fabric/+/31856
guoger (Wed, 19 Jun 2019 13:54:36 GMT):
@tock i'll get to your CR stack tomorrow (i don't have +2 permission yet...)
tock (Thu, 20 Jun 2019 06:05:56 GMT):
@guoger Congratulations! I left you a basket of cherries as a welcome present ;-)
tock (Thu, 20 Jun 2019 06:05:56 GMT):
@guoger Congratulations! I left you a basket of cherries as a promotion present ;-)
guoger (Thu, 20 Jun 2019 06:09:09 GMT):
really appreciated!! i just need to find my way to Haifa now :thinking:
yacovm (Thu, 20 Jun 2019 06:23:22 GMT):
I think he meant cherry picks, but not sure
tock (Thu, 20 Jun 2019 08:47:12 GMT):
I picked them myself
jyellick (Wed, 26 Jun 2019 04:37:36 GMT):
@adarshsaraf123 Thanks for getting those CRs for the config processing consolidation put together, but it looks like they're now in conflict ( https://gerrit.hyperledger.org/r/c/fabric/+/30811 ). @guoger mentioned that you were planning to rebase them, but just wanted to confirm.
jyellick (Wed, 26 Jun 2019 04:38:54 GMT):
@tock Were you planning to cherry-pick https://gerrit.hyperledger.org/r/c/fabric/+/31856 and https://gerrit.hyperledger.org/r/c/fabric/+/31932 or would you like me to?
adarshsaraf123 (Wed, 26 Jun 2019 04:42:52 GMT):
Yes @jyellick. I will do the rebasing.
adarshsaraf123 (Wed, 26 Jun 2019 04:42:52 GMT):
Yes @jyellick I will do the rebasing.
tock (Wed, 26 Jun 2019 06:19:49 GMT):
@jyellick I will cherry-pick them.
adarshsaraf123 (Wed, 26 Jun 2019 11:12:07 GMT):
@jyellick @guoger and others, The rebasing of the Consensus MetadataValidator CRs is done. Please review and hopefully we can get it merged soon.
guoger (Mon, 08 Jul 2019 11:13:54 GMT):
@yacovm looking at your [comment](https://gerrit.hyperledger.org/r/c/fabric/+/32206/2/orderer/consensus/etcdraft/chain.go#355)
> why can't we just piggyback on the messages sent from the leader? then we don't need this whole periodic sending because we can just grab the latest metadata and send it.
which metadata are you referring to?
minollo (Mon, 08 Jul 2019 11:17:09 GMT):
@yacovm, @dave.enyeart Is there any activity going on about https://jira.hyperledger.org/browse/FAB-5288? Is it actually being rolled out in 2.0? What about backporting it to 1.4.x LTS as well?
yacovm (Mon, 08 Jul 2019 19:59:03 GMT):
what do you mean what metadata? you literally added some new metadata, no?
guoger (Tue, 09 Jul 2019 01:59:17 GMT):
pls see comment in that CR
guoger (Fri, 12 Jul 2019 09:13:02 GMT):
@yacovm @jyellick let’s chat here. it’s all about what we want to show while no leader is elected. I think this metrics will be more useful if it shows how many nodes can be reached in this situation.
guoger (Fri, 12 Jul 2019 09:14:02 GMT):
If that’s not the case, then you guys are definitely right, we could grab the progress from leader and disseminate
yacovm (Fri, 12 Jul 2019 09:55:56 GMT):
i thought you did that thing to know when you have enough standby nodes? @guoger
yacovm (Fri, 12 Jul 2019 09:56:04 GMT):
not to know the status when you have no leader
yacovm (Fri, 12 Jul 2019 09:56:09 GMT):
that's what you said back then
guoger (Fri, 12 Jul 2019 09:57:44 GMT):
What’s your definition of standby?
guoger (Fri, 12 Jul 2019 09:58:41 GMT):
It’s likely that my words there were not precise enough
yacovm (Fri, 12 Jul 2019 09:59:31 GMT):
follower
guoger (Fri, 12 Jul 2019 10:06:30 GMT):
It’s not about knowing *enough* nodes. It’s about knowing exactly how many nodes, even when there’s not quorum
jyellick (Fri, 12 Jul 2019 13:42:29 GMT):
@guoger I'm really quite confused by this. My understanding was that our primary motivator for this CR was to be able to tell admins whether or not a config update would break cluster quorum. So, necessarily, a quorum does exist, else, the config update would be rejected as no leader is available anyway.
yacovm (Fri, 12 Jul 2019 13:44:50 GMT):
^ that's what I also inferred last time
yacovm (Fri, 12 Jul 2019 13:45:04 GMT):
but instead config update - a maintenance period
yacovm (Fri, 12 Jul 2019 13:45:43 GMT):
but that raises the question - if multiple people wanna do a config update / maintenance, at the same time, without synchronizing - it can be problematic
yacovm (Fri, 12 Jul 2019 13:45:51 GMT):
so they might as well just always be forced to synchronize
guoger (Fri, 12 Jul 2019 13:48:57 GMT):
So, i think there are 2 motivators:
1) display alive nodes in cluster
2) and as what you said there, reject dangerous reconfig (quorum loss)
and i think the main confusion is around 1).
guoger (Fri, 12 Jul 2019 13:50:30 GMT):
and i'm lost when you say maintenance... could you elaborate?
yacovm (Fri, 12 Jul 2019 13:50:52 GMT):
like changing a router/switch
guoger (Fri, 12 Jul 2019 13:51:42 GMT):
oh, i thought you were talking about maintenance mode for migration...
guoger (Fri, 12 Jul 2019 13:53:50 GMT):
basically i think they should synchronize on when to do config update and maintenance. But, that's orthogonal to this specific task, no?
jyellick (Fri, 12 Jul 2019 14:01:59 GMT):
My concern with (1) is that we can't really do it reliably. We're having to implement it on our own, and, unless there's a quorum, we can't say with any certainty whether that number has any bearing on the reality of the network.
jyellick (Fri, 12 Jul 2019 14:03:35 GMT):
Knowing the number of 'alive nodes' doesn't seem very useful by itself. As having 100% alive nodes doesn't mean that the network can form quorum
jyellick (Fri, 12 Jul 2019 14:03:51 GMT):
(if you allow one-way network partitioning, for instance)
yacovm (Fri, 12 Jul 2019 14:04:39 GMT):
my general argument against doing it back then, is that we don't gain much from the ability of letting all administrators see the communication between the leader and the nodes of the cluster
yacovm (Fri, 12 Jul 2019 14:04:54 GMT):
but we pay a lot in complexity
yacovm (Fri, 12 Jul 2019 14:07:11 GMT):
let me ask you this
yacovm (Fri, 12 Jul 2019 14:07:31 GMT):
we log the connection errors anyway, right?
guoger (Fri, 12 Jul 2019 14:08:20 GMT):
First of all, let me make sure we are on the same page: @yacovm is against the story (adding "alive node" metrics), and @jyellick is arguing about what this metrics should actually mean, correct?
yacovm (Fri, 12 Jul 2019 14:09:28 GMT):
i just don't know what an administrator of org A is going to do with information that org B cannot connect to the leader node
yacovm (Fri, 12 Jul 2019 14:09:38 GMT):
since the leader node sends hearbeats anyway to everyone
jyellick (Fri, 12 Jul 2019 14:10:42 GMT):
My initial goal/understanding, was that we were simply going to take the active node information, maintained by etcd, at the leader, and disseminate it to the followers. In this way, we could make smart decisions about config updates that break quorum, and, be able to uniformly report via metrics the cluster health (in terms of nodes in quorum) through every orderer's metrics.
yacovm (Fri, 12 Jul 2019 14:12:55 GMT):
I think that if we do that, we should just send that `Status()` result from the leader to the nodes, and not collect the information ourselves based on the last received timestamp
yacovm (Fri, 12 Jul 2019 14:13:01 GMT):
(that's what I said in the CR)
guoger (Fri, 12 Jul 2019 14:14:39 GMT):
@yacovm what is this question about?
> we log the connection errors anyway, right?
yacovm (Fri, 12 Jul 2019 14:15:27 GMT):
so what i mean is - I don't think this is very useful, but if we do it, we should do it with as less code as possibl
yacovm (Fri, 12 Jul 2019 14:15:27 GMT):
so what i mean is - I don't think this is very useful, but if we do it, we should do it with as less code as possible and just take what the leader knows anyway
yacovm (Fri, 12 Jul 2019 14:16:10 GMT):
i'm saying that if people monitor logs, then they should see something is wrong
guoger (Fri, 12 Jul 2019 14:16:51 GMT):
OK, we can probably talk about `I don't think this is very useful` later, since you seem to be ok with `if we do it`
yacovm (Fri, 12 Jul 2019 14:18:48 GMT):
I don't mean to belittle or anything, I'm just saying this isn't essential
jyellick (Fri, 12 Jul 2019 14:19:53 GMT):
If we can accomplish this without injecting a lot of new code and complexity, the value seems there to me.
jyellick (Fri, 12 Jul 2019 14:20:27 GMT):
But as mentioned, I had assumed we'd simply be disseminating the active node info etcd collects from the leader to the followers. (Which is hopefully low complexity)
guoger (Fri, 12 Jul 2019 14:21:53 GMT):
my argument is simply: suppose we have an outage of nodes, only 2/5 are alive. I think this metrics is more helpful if it shows 2. Otherwise, there's really not much info with `activeNodes = 0`
yacovm (Fri, 12 Jul 2019 14:22:33 GMT):
hmmm, i wonder what stats are really collected from the leader when the quorum is lost?
yacovm (Fri, 12 Jul 2019 14:22:36 GMT):
have you checked?
guoger (Fri, 12 Jul 2019 14:23:02 GMT):
there's no leader when quorum is lost
yacovm (Fri, 12 Jul 2019 14:23:10 GMT):
right
yacovm (Fri, 12 Jul 2019 14:23:16 GMT):
so what does the previous leader print?
guoger (Fri, 12 Jul 2019 14:23:34 GMT):
print? do you mean repeated campaign?
yacovm (Fri, 12 Jul 2019 14:23:55 GMT):
no i mean from that API call that Jason recommended
guoger (Fri, 12 Jul 2019 14:25:00 GMT):
if it's not leader, it only returns the info about logs
guoger (Fri, 12 Jul 2019 14:25:08 GMT):
index etc
jyellick (Fri, 12 Jul 2019 14:25:20 GMT):
If there are only 2/5 nodes alive, then we return that we have no quorum, so we can't give network state. This seems fine to me.
jyellick (Fri, 12 Jul 2019 14:26:46 GMT):
As I said before, for instance, if network latencies suddenly climb to be huge, all nodes might be 'active', but quorum would still be incapable of forming. There are a lot of edge cases and nuances that etcdraft handles and tracks for us, that we don't want to handle.
yacovm (Fri, 12 Jul 2019 14:27:31 GMT):
we have `msg_send_time` in the cluster metrics btw ;)
yacovm (Fri, 12 Jul 2019 14:27:43 GMT):
if network is getting clogged this would go up
guoger (Fri, 12 Jul 2019 14:30:42 GMT):
> If there are only 2/5 nodes alive, then we return that we have no quorum, so we can't give network state. This seems fine to me.
ok, i think we can actually provide admin with more info, but since we clearly have a vote 2 vs 1, i'll make the change.
FWIW, i think `msg_send_time >= election timeout` and `active node >= quorum`, admin should have a better chance to diagnose problem :P
jyellick (Fri, 12 Jul 2019 14:31:18 GMT):
@guoger @yacovm could you re-+2 this one? https://gerrit.hyperledger.org/r/c/fabric/+/32278 I quick added a way to override for the peer too
jyellick (Fri, 12 Jul 2019 14:32:38 GMT):
@guoger I don't disagree that there's some value, tracking the other replicas that a node has seen, and presenting this value to the user. But given the amount of additional code required, especially adding yet another tick based tracker, the value doesn't seem there for me.
jyellick (Fri, 12 Jul 2019 14:32:38 GMT):
@guoger I don't disagree that there's some value, tracking the other replicas that a node has seen, and presenting this value to the user. But given the amount of additional code required, especially adding yet another tick based tracker, the value to complexity doesn't seem there for me.
yacovm (Fri, 12 Jul 2019 14:32:51 GMT):
> ok, i think we can actually provide admin with more info, but since we clearly have a vote 2 vs 1, i'll make the change.
I'm not voting against, I'm voting abstain ! :)
guoger (Fri, 12 Jul 2019 14:35:09 GMT):
hmm, i fear we still need a ticker to periodically get this info. If we send Status along with every msg, there might be a performance penalty
guoger (Fri, 12 Jul 2019 14:35:22 GMT):
(`Status()` makes a copy of map
jyellick (Fri, 12 Jul 2019 14:36:23 GMT):
Could we only send status when it changes?
jyellick (Fri, 12 Jul 2019 14:36:35 GMT):
I suppose for new nodes that would be a problem
guoger (Fri, 12 Jul 2019 14:36:45 GMT):
right, or if some msg got lost
guoger (Fri, 12 Jul 2019 14:37:01 GMT):
some nodes may never have up-to-date info, if status doesn't change
jyellick (Fri, 12 Jul 2019 14:39:10 GMT):
I suppose a ticker may be unavoidable then, but still, not tracking it ourselves I would expect simplifies things
guoger (Fri, 12 Jul 2019 14:40:06 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-orderer-dev?msg=nJZxNiBinuhdskN44) lol, you are making this decision harder :P
guoger (Fri, 12 Jul 2019 14:40:55 GMT):
@jyellick i have another topic to discuss: where do we check quorum loss against reconfig, as you commented in https://gerrit.hyperledger.org/r/c/fabric/+/32221
guoger (Fri, 12 Jul 2019 14:41:37 GMT):
i think if we move `ValidateMetadata` from `Consenter` to `Chain`, it would simplifies things a lot. I uploaded a draft patch for this: https://gerrit.hyperledger.org/r/c/fabric/+/32284
guoger (Fri, 12 Jul 2019 14:42:09 GMT):
basically moving from current _stateless_ validation to be _stateful_
guoger (Fri, 12 Jul 2019 14:42:51 GMT):
all metadata parsing will still be done at one place
jyellick (Fri, 12 Jul 2019 14:48:18 GMT):
Let me take a look
guoger (Fri, 12 Jul 2019 14:48:54 GMT):
thx
jyellick (Fri, 12 Jul 2019 14:53:06 GMT):
I think the approach generally makes sense. If it's not too challenging, I'd split it into two CRs though, one moving it from consenter to chain, and the other adding the additional checks.
guoger (Fri, 12 Jul 2019 14:58:22 GMT):
absolutely, that CR was just intended to show the approach (that's why i -1'ed it)
guoger (Fri, 12 Jul 2019 14:59:03 GMT):
OK, i'll update the CR. thx for your time :)
yacovm (Fri, 12 Jul 2019 19:19:42 GMT):
we don't really need a ticker. we can track the last time we sent a message to each node without a ticker, and send the message every 10 seconds.
yacovm (Fri, 12 Jul 2019 19:20:10 GMT):
this way we can piggyback on existing messages from the leader
jyellick (Fri, 12 Jul 2019 19:23:55 GMT):
That seems reasonable to me, what do you think @guoger ?
guoger (Sat, 13 Jul 2019 00:27:13 GMT):
What do you mean by every 10 seconds without a ticker?
guoger (Sat, 13 Jul 2019 00:27:13 GMT):
What do you mean by every 10 seconds without a ticker? could you elaborate? @jyellick @yacovm
jyellick (Sat, 13 Jul 2019 01:00:55 GMT):
The leader necessarily sends a message at least every heartbeat interval. So, if the last metadata message was sent more than 10s ago, piggy back it
guoger (Sat, 13 Jul 2019 01:08:49 GMT):
Oh, then we need to have a tracker to record last time a metadata is sent, furthermore we still need to periodically invoke Status to get latest info. And we still need to toggle it when leadership is changed. I’m not sure this is actually simpler... wdyt
guoger (Sat, 13 Jul 2019 01:08:49 GMT):
Oh, then we need to have a tracker to record last time a metadata is sent, furthermore we still need to invoke Status to get latest info, and try to avoid duplicate here when multiple expires happen. And we still need to toggle it when leadership is changed. I’m not sure this is actually simpler... wdyt
guoger (Sat, 13 Jul 2019 01:10:17 GMT):
I don’t think piggyback buys us a lot, we don’t have much message overhead for metadata anyway - once per election timeout, which is 10s by default.
guoger (Sat, 13 Jul 2019 01:15:47 GMT):
Btw, I think we should have a metrics Has_Leader, or Leader_ID.
jyellick (Mon, 15 Jul 2019 03:01:11 GMT):
I'd lean towards whatever is simpler to implement. I would hope either of them would be fairly straightforward. In the case of piggy-backing it seems as easy as "If the status hasn't been refreshed in the last 10 seconds, refresh it. If the node I'm sending the message to hasn't gotten the status in the last 10 seconds, piggy-back it.", which I would think would just be a couple instance vars and a few lines of code, but maybe I'm underestimating it.
guoger (Mon, 15 Jul 2019 05:25:28 GMT):
yet another factor is, "if i'm leader, disseminate status/if i'm follower, waiting for status/if i'm leaderless, reset metrics", which is less straightforward if we go down this path.
guoger (Mon, 15 Jul 2019 05:30:52 GMT):
which is also less testable - essentially we have extra N+1 fake clock to advance
guoger (Mon, 15 Jul 2019 05:30:52 GMT):
which is also less testable - essentially we have extra N+1 fake clock to advance. cc @jyellick
minollo (Mon, 15 Jul 2019 12:32:19 GMT):
@dave.enyeart , anyone else?
PulkitSarraf (Tue, 16 Jul 2019 09:56:02 GMT):
Has joined the channel.
PulkitSarraf (Tue, 16 Jul 2019 09:56:03 GMT):
Anyone added an orderer using etcdraft protocol in a running network
i am getting an error when i modify the config.json file and update the channel with new orderer address and its tls certs
083 Deactivating node 6 in channel firstchannel with endpoint of orderer6.example.com:7050 due to TLS certificate change
yacovm (Tue, 16 Jul 2019 10:01:17 GMT):
it's not necessarily a bad thing
guoger (Tue, 16 Jul 2019 10:31:49 GMT):
@PulkitSarraf also, pls do not cross-post question. this channel is used for development related discussion.
PulkitSarraf (Tue, 16 Jul 2019 10:32:34 GMT):
I am sorry for that
Utsav_Solanki (Fri, 19 Jul 2019 04:27:18 GMT):
Has joined the channel.
Utsav_Solanki (Fri, 19 Jul 2019 04:27:20 GMT):
i am getting in docker logs of orderer as --> orderer.AtomicBroadcast grpc.method=Deliver grpc.peer_address= ipxx x x x :port x error="context finished before block retrieved: context canceled" grpc.code=Unknown grpc.call_duration=5h48m38.655538305s
is it error or normal behavior?
guoger (Fri, 26 Jul 2019 11:04:44 GMT):
Re your comment in JIRA (hard to access it at the moment): we are doing post-consensus transfer, because it’s more complicated to do pre-consensus transfer - in which case we need to block incoming normal tx, wait for the success of leader transfer and proceed. With post-consensus, we simply make an attempt, if it fails, that’s fine since eventually a new node can start campaign anyway, so we don’t really care about the success of transfer. @yacovm
yacovm (Fri, 26 Jul 2019 11:21:42 GMT):
@guoger so you're saying, the leader simply gives up its leadership and some node picks it up?
guoger (Fri, 26 Jul 2019 11:22:39 GMT):
Yes.
yacovm (Fri, 26 Jul 2019 11:22:51 GMT):
and the config change happens *before* or *after* ?
yacovm (Fri, 26 Jul 2019 11:22:57 GMT):
the config change that removes the leader
guoger (Fri, 26 Jul 2019 11:23:08 GMT):
Before
yacovm (Fri, 26 Jul 2019 11:23:15 GMT):
why?
guoger (Fri, 26 Jul 2019 11:24:19 GMT):
For the reason i mentioned above: mostly for the sake of simplicity
yacovm (Fri, 26 Jul 2019 11:26:05 GMT):
i don't understand the point of doing a post consensus transfer
yacovm (Fri, 26 Jul 2019 11:26:32 GMT):
since we do it in 2 steps always
yacovm (Fri, 26 Jul 2019 11:27:04 GMT):
if you remove the leader in any sized cluster, isn't that problematic?
yacovm (Fri, 26 Jul 2019 11:28:25 GMT):
do you remember which config change we do first? the fabric or the raft one? I think it's the fabric one, no?
guoger (Fri, 26 Jul 2019 11:40:19 GMT):
Yes, the fabric one. Removing leader is not a problem in cluster larger than 2 nodes, since there always will be a new leader being elected if transfer fails.
guoger (Fri, 26 Jul 2019 11:43:57 GMT):
Other than the reason above, even if we do pre-consensus transfer, this node might still be elected as leader after a network crash, and we end up with the same situation
yacovm (Fri, 26 Jul 2019 12:15:39 GMT):
oh i actually remember now that we had this mechanism where the new leader can detect the fabric config change and then propose only the raft config change, right?
guoger (Fri, 26 Jul 2019 12:26:33 GMT):
That’s right
huxd (Fri, 02 Aug 2019 01:33:34 GMT):
Has joined the channel.
SatheeshNehru (Mon, 12 Aug 2019 05:21:18 GMT):
Has joined the channel.
kostas (Mon, 12 Aug 2019 21:23:55 GMT):
As I'm getting back into a semi-regular schedule, feel free to tag me on any orderer-related PRs, if there are any. (I understand development on that front has quieted down quite a bit.)
dave.enyeart (Wed, 14 Aug 2019 11:06:54 GMT):
There's a request to extend Raft documentation: https://jira.hyperledger.org/browse/FAB-16306 FYI @pandrejko @joe-alewine
pandrejko (Wed, 14 Aug 2019 11:06:54 GMT):
Has joined the channel.
joe-alewine (Wed, 14 Aug 2019 13:50:09 GMT):
@dave.enyeart If the process is known (I assume it's written down in a design doc somewhere) and can be tested, I'm all for creating a tutorial (preferably not one based on BYFN, given recent discussions) describing how to do it
conanoc (Wed, 21 Aug 2019 09:04:31 GMT):
Has joined the channel.
guoger (Thu, 22 Aug 2019 07:16:46 GMT):
has anyone seen the error in FAB-16373 ?
```
2019-08-21 09:17:28.019 UTC [orderer.consensus.kafka] processTimeToCut -> DEBU 989 [channel: fatico-dedicated] It's a time-to-cut message for block [15]
2019-08-21 09:17:28.019 UTC [orderer.consensus.kafka] processTimeToCut -> DEBU 98a [channel: fatico-dedicated] Ignoring stale time-to-cut-message for block [15]
2019-08-21 09:17:28.019 UTC [orderer.consensus.kafka] processMessagesToBlocks -> DEBU 98b [channel: fatico-dedicated] Successfully unmarshalled consumed message, offset is 64. Inspecting type...
2019-08-21 09:17:28.019 UTC [orderer.consensus.kafka] processTimeToCut -> DEBU 98c [channel: fatico-dedicated] It's a time-to-cut message for block [18]
2019-08-21 09:17:28.019 UTC [orderer.consensus.kafka] processMessagesToBlocks -> WARN 98d [channel: fatico-dedicated] got larger time-to-cut message (18) than allowed/expected (16) - this might indicate a bug
2019-08-21 09:17:28.019 UTC [orderer.consensus.kafka] processMessagesToBlocks -> ERRO 98e [channel: fatico-dedicated] Consenter for channel exiting
```
it looks very strange to me...
also, I think we should panic there, instead of exiting chain (we have a TODO in the code for this)
guoger (Thu, 22 Aug 2019 07:17:16 GMT):
and a side question: how much energy do we want to spend in maintaining kafka? are we planning to sunset it in 2.1?
shitaibin (Thu, 22 Aug 2019 12:17:09 GMT):
Has joined the channel.
jyellick (Thu, 22 Aug 2019 16:04:39 GMT):
@guoger I've seen this error generally when the Kafka logs are corrupted.
jyellick (Thu, 22 Aug 2019 16:04:53 GMT):
Usually this is because everything's on one machine, and they kill everything at once.
jyellick (Thu, 22 Aug 2019 16:05:08 GMT):
The orderers use fsync, the brokers do not, so they end up ahead of the brokers.
jyellick (Thu, 22 Aug 2019 16:06:09 GMT):
If one orderer is further ahead than the others, then its block height will be higher, and it will send time to cut messages exhibiting this pattern.
jyellick (Thu, 22 Aug 2019 16:09:22 GMT):
You'll see this especially if you re-bootstrap an orderer.
guoger (Thu, 22 Aug 2019 16:31:32 GMT):
@jyellick ah, good to know... thx, could you kindly post in FAB-16373 and close it?
jyellick (Thu, 22 Aug 2019 16:31:52 GMT):
Sure
kostas (Thu, 22 Aug 2019 17:09:07 GMT):
Great. I was wondering how this could have happened in a normal environment.
kostas (Thu, 22 Aug 2019 17:10:19 GMT):
RE: maintaining Kafka — my take is that it should be sunsetted as fast as possible. 2.0 would be a natural cutoff point for this, but perhaps this is a bit too aggressive.
kostas (Thu, 22 Aug 2019 17:11:20 GMT):
Has IBP switched to Raft? If they don, I'm curious to see what the feedback on that front is.
kostas (Thu, 22 Aug 2019 17:11:20 GMT):
Has IBP switched to Raft? If they have, I'm curious to see what the feedback on that front is.
Utsav_Solanki (Tue, 17 Sep 2019 07:45:21 GMT):
in my orderer2 i am getting this error logs
Utsav_Solanki (Tue, 17 Sep 2019 07:45:56 GMT):
2019-09-13 16:26:17.503 UTC [orderer.consensus.kafka] processMessagesToBlocks -> ERRO 08c [channel: mychannel] Error during consumption: kafka: error while consuming channel/0: dial tcp: lookup kafka0 on 127.0.0.11:53: no such host
2019-09-13 16:26:17.503 UTC [orderer.consensus.kafka] processMessagesToBlocks -> WARN 08d [channel: mychannel] Deliver sessions will be dropped if consumption errors continue.
2019-09-13 16:26:17.504 UTC [orderer.consensus.kafka] processMessagesToBlocks -> ERRO 08e [channel: testchainid] Error during consumption: kafka: error while consuming testchainid/0: dial tcp: lookup kafka0 on 127.0.0.11:53: no such host
2019-09-13 16:26:17.504 UTC [orderer.consensus.kafka] processMessagesToBlocks -> WARN 08f [channel: testchainid] Deliver sessions will be dropped if consumption errors continue.
2019-09-13 16:26:19.506 UTC [orderer.consensus.kafka] processMessagesToBlocks -> WARN 090 [channel: testchainid] Consumption will resume.
guoger (Tue, 17 Sep 2019 11:18:32 GMT):
pls ask question in #fabric-orderer , this channel is for dev only. as for your error, depends on how you provision the network, you'll need to configure the system to be able to resolve `kafka0` to correct ip (docker-compose does this automatically)
Utsav_Solanki (Tue, 17 Sep 2019 11:38:02 GMT):
ok
jyellick (Tue, 24 Sep 2019 19:38:42 GMT):
@guoger @yacovm This may be well understood behavior, but it is already causing us problems. When a node is removed from a raft consenter set, it will not rejoin if re-added, because `TrackChain` is never invoked for that chain, because the node is in the consenter set at startup.
jyellick (Tue, 24 Sep 2019 19:40:25 GMT):
I would propose, that when we detect we are being removed, and initiate shutdown of the consenter, that we invoke the `TrackChain` API so that we are put into the same state as if we had just been started, but were not in the consenter set.
jyellick (Tue, 24 Sep 2019 19:40:39 GMT):
Are there any objections or concerns to this?
jyellick (Tue, 24 Sep 2019 19:41:44 GMT):
(FWIW, as tooling emerges to automate the addition and removal of nodes from the consenter sets, it's fairly inevitable that someone will be playing with a set of consenters, and remove and re-add the same consenter without restarting it. This seems like a perfectly common usage pattern, so it seems like something we should support).
yacovm (Tue, 24 Sep 2019 20:38:39 GMT):
> it's fairly inevitable that someone will be playing with a set of consenters, and remove and re-add the same consenter without restarting it
Very mature... it's essentially like playing with your new phone, right?
> but it is already causing us problems.
Can you point me to a case where it caused a problem? When we were designing this I was sure that we removed consenters only if you have a full membership and want to scale down for good, or if you just want to evict an OSN from a channel.
> Are there any objections or concerns to this?
The code assumes it's a genesis block, so it won't work.... you are free to try and see for yourself or prove me wrong
yacovm (Tue, 24 Sep 2019 20:38:39 GMT):
> it's fairly inevitable that someone will be playing with a set of consenters, and remove and re-add the same consenter without restarting it
Very mature... it's essentially like playing with your new phone, right?
> but it is already causing us problems.
Can you point me to a case where it caused a problem? When we were designing this I was sure that we removed consenters only if you have a full membership and want to scale down for good, or if you just want to evict an OSN from a channel.
> Are there any objections or concerns to this?
The code assumes it's a genesis block, so it won't work.... you are free to try and see for yourself or prove me wrong
yacovm (Tue, 24 Sep 2019 20:38:39 GMT):
> it's fairly inevitable that someone will be playing with a set of consenters, and remove and re-add the same consenter without restarting it
Very mature... it's essentially like playing with your new phone, right?
> but it is already causing us problems.
Can you point me to a case where it caused a problem? When we were designing this I was sure that we removed consenters only if you have a full membership and want to scale down for good, or if you just want to evict an OSN from a channel.
> Are there any objections or concerns to this?
The code assumes it's a genesis block, so it won't work.... you are free to try and see for yourself or prove me wrong
yacovm (Tue, 24 Sep 2019 20:38:39 GMT):
> it's fairly inevitable that someone will be playing with a set of consenters, and remove and re-add the same consenter without restarting it
Very mature... it's essentially like playing with your new phone, right?
> but it is already causing us problems.
Can you point me to a case where it caused a problem? When we were designing this I was sure that we removed consenters only if you have a full membership and want to scale down for good, or if you just want to evict an OSN from a channel.
> Are there any objections or concerns to this?
The code assumes it's a genesis block, so it won't work.... you are free to try and see for yourself or prove me wrong
yacovm (Tue, 24 Sep 2019 20:38:43 GMT):
@jyellick ^
yacovm (Tue, 24 Sep 2019 20:39:31 GMT):
Of course, I'm not saying we can't massage the code to make it work, it's just that i think it involves more than that.
jyellick (Tue, 24 Sep 2019 20:49:25 GMT):
> Can you point me to a case where it caused a problem?
Already gotten bug reports from two different users.
yacovm (Tue, 24 Sep 2019 20:50:25 GMT):
ugh, pesky users
jyellick (Tue, 24 Sep 2019 20:52:10 GMT):
> The code assumes it's a genesis block, so it won't work.... you are free to try and see for yourself or prove me wrong
I noticed it seemed to expect a genesis block, I was hoping it was as generic as a 'config block', but perhaps not.
jyellick (Tue, 24 Sep 2019 20:52:15 GMT):
What happens if it dies mid-replication?
yacovm (Tue, 24 Sep 2019 20:52:54 GMT):
what dies?
jyellick (Tue, 24 Sep 2019 20:54:11 GMT):
If the orderer starts up, is not in a channel, and then the TrackChain recognizes that the orderer has been added, but crashes before the blockchain has been replicated up to the point where the orderer is added?
jyellick (Tue, 24 Sep 2019 20:54:52 GMT):
I would assume that the orderer would start, the length of the chain would be non-zero, the orderer would still not be a consenter for that chain, and, it would somehow resume.
yacovm (Tue, 24 Sep 2019 20:55:40 GMT):
FAB-13552
yacovm (Tue, 24 Sep 2019 20:55:44 GMT):
check it out @jyellick ^
jyellick (Tue, 24 Sep 2019 20:56:00 GMT):
Thanks, got to run, but will look at it later tonight
yacovm (Tue, 24 Sep 2019 20:56:55 GMT):
> I would assume that the orderer would start, the length of the chain would be non-zero, the orderer would still not be a consenter for that chain, and, it would somehow resume.
yeah, we can handle crashes in the middle obviously. Even crashes during initial onboarding.
jyellick (Wed, 25 Sep 2019 14:27:46 GMT):
@yacovm I've been walking that code, and I'm really not seeing why using any config block would not work.
jyellick (Wed, 25 Sep 2019 14:28:02 GMT):
It seems very similar to the path of crashing during onboarding.
jyellick (Wed, 25 Sep 2019 14:28:50 GMT):
The replicator starts up, tries to append the block, but realizes the ledger height is further, so skips it. Then it probes the channels to see if it should replicate, and if so, does.
yacovm (Wed, 25 Sep 2019 14:41:04 GMT):
`ChannelCreationBlockToGenesisBlock`
jyellick (Wed, 25 Sep 2019 14:42:25 GMT):
I don't see that it would return an error?
yacovm (Wed, 25 Sep 2019 14:47:15 GMT):
ah so @jyellick I think that `TrackChain` is always called with GB right?
yacovm (Wed, 25 Sep 2019 14:47:19 GMT):
currently, that is?
yacovm (Wed, 25 Sep 2019 14:47:26 GMT):
even if the height is bigger?
yacovm (Wed, 25 Sep 2019 14:48:44 GMT):
I think we should call it with the GB as well and not with a higher block
jyellick (Wed, 25 Sep 2019 14:49:27 GMT):
As best as I can tell... if the block number is already committed, whether it's GB, or any other block, it's essentially ignored?
jyellick (Wed, 25 Sep 2019 14:49:39 GMT):
Otherwise I think there would be a real bug
jyellick (Wed, 25 Sep 2019 14:49:57 GMT):
The `ChannelCreationBlockToGenesisBlock` expects a block from the system channel, which embeds the contents for the genesis block of the channel.
jyellick (Wed, 25 Sep 2019 14:50:18 GMT):
We're passing in the genesis block for the channel, and attempting to extract its own block from there.
jyellick (Wed, 25 Sep 2019 14:50:30 GMT):
Fortunately `appendBlock` is always a no-op, because the block is already committed.
jyellick (Wed, 25 Sep 2019 14:51:03 GMT):
I would think the right way to do it, would be to pass in a nil block, and if it's nil, then say the chain already exists.
jyellick (Wed, 25 Sep 2019 14:51:03 GMT):
I would think the right way to do it, would be to pass in a nil block, and if it's nil, then say the chain already exists, skip trying to append the genesis block.
jyellick (Wed, 25 Sep 2019 14:51:38 GMT):
As best as I can tell, appending the genesis block is the only thing that block i used for? (And, again, in the 'inactive chain' case, it's always ignored)
yacovm (Wed, 25 Sep 2019 14:52:57 GMT):
ah so i missed this: `block.Header.Number = 0`
yacovm (Wed, 25 Sep 2019 14:53:13 GMT):
This is why it always works, heh
yacovm (Wed, 25 Sep 2019 14:53:43 GMT):
so you can try it and see if it works....
jyellick (Wed, 25 Sep 2019 14:55:50 GMT):
Yes, I expect that I will. I also think I might make the change to allow the nil block. Passing in the genesis block is very misleading, as it really must be the channel creation block in order to work as intended.
yacovm (Wed, 25 Sep 2019 15:06:11 GMT):
@jyellick just one rule.... no Ginkgo.... ;)
jyellick (Wed, 25 Sep 2019 15:07:12 GMT):
But I was really looking forward to spending a week rewriting the whole test suite :slight_smile:
jyellick (Fri, 04 Oct 2019 15:46:20 GMT):
@kostas @guoger We've been observing an issue when producing large transactions to Kafka, that we're seeing those messages get duplicated. As best as I can tell, when retries are enabled, message duplication is an expected behavior of Kafka. However, introduced in v0.11 they allow "exactly once" semantics. I _think_, turning it on would be as easy as flipping https://github.com/Shopify/sarama/blob/master/config.go#L165-L167 to true. I was wondering if you guys ever looked at this setting, and if there's a reason why we left it false?
jyellick (Mon, 07 Oct 2019 14:43:28 GMT):
@guoger @kostas ^ ping?
guoger (Tue, 08 Oct 2019 06:32:48 GMT):
sorry for late response, was out of office for holiday here... i'm afraid that code was written before i joined project but AFAICT flipping `idempotent` along with some [other configs](https://github.com/Shopify/sarama/commit/8e2b04b363c226275dbf10d771fb044e387bbd75#diff-b4bda758a2aef091432646c354b4dc59R517-R531) could solve the problem.
guoger (Tue, 08 Oct 2019 06:34:36 GMT):
although, ultimately we still cannot provide end-to-end guarantee to fabric clients that a tx is failed to commit, right?
guoger (Tue, 08 Oct 2019 06:37:48 GMT):
if a msg is *seemingly* failed to be committed, and the error is returned to fabric clients, there's no way to prevent clients from resubmitting that tx, which results in duplicate. (basically we need to introduce our own seq no to ensure exactly-once semantics on fabric side, similar to kafka)
yacovm (Tue, 08 Oct 2019 09:25:10 GMT):
@guoger - Fabric already has exactly once semantics for transactions. The issue Jason is complaining about is that for big transactions - I guess the Kafka client re-submits the transaction which results in the transaction going twice into a block or twice in different blocks, and that's just a waste of space.
yacovm (Tue, 08 Oct 2019 09:26:29 GMT):
In any case, any client can always record a transaction and spam it to the orderer, even if that transaction isn't made by him.
To prevent that, we need to have epochs but it's not implemented currently, I guess because it makes the SDK's life more difficult.
yacovm (Tue, 08 Oct 2019 09:26:42 GMT):
we do have an epoch field in the transaction as you know, but it's always 0 ;)
guoger (Tue, 08 Oct 2019 09:29:50 GMT):
i think fabric has `most-once` instead of `exactly-once` - if client submits same tx several times to orderer, it only applies one
guoger (Tue, 08 Oct 2019 09:30:49 GMT):
and we need the `epoch` you mentioned here to provide an end-to-end `exactly-once` semantics, no?
yacovm (Tue, 08 Oct 2019 09:31:38 GMT):
By Fabric I meant the validation logic ;)
yacovm (Tue, 08 Oct 2019 09:31:56 GMT):
so yeah - exactly once, most once - depends on where in the flow you look
yacovm (Tue, 08 Oct 2019 09:32:04 GMT):
> and we need the epoch you mentioned here to provide an end-to-end exactly-once semantics, no?
No...
yacovm (Tue, 08 Oct 2019 09:32:10 GMT):
that's not what I meant
yacovm (Tue, 08 Oct 2019 09:32:13 GMT):
let me elaborate
yacovm (Tue, 08 Oct 2019 09:32:36 GMT):
when you send a transaction with an ID that was committed before, it is ignored
yacovm (Tue, 08 Oct 2019 09:32:44 GMT):
however the transaction will still get into the raw ledger
yacovm (Tue, 08 Oct 2019 09:32:49 GMT):
so in a sense, this is pollution
yacovm (Tue, 08 Oct 2019 09:33:09 GMT):
a malicious client can take a transaction of any client it wishes and spam orderers
yacovm (Tue, 08 Oct 2019 09:33:36 GMT):
even if that client is revoked
yacovm (Tue, 08 Oct 2019 09:33:54 GMT):
the way to prevent it, is to use the epoch field in the transaction
yacovm (Tue, 08 Oct 2019 09:34:17 GMT):
and make the orderers reject transactions with epoch which is too far in the past, and increment the epoch once in a while
yacovm (Tue, 08 Oct 2019 09:35:10 GMT):
however it would also mean the SDK would need to see what is the current epoch.
yacovm (Tue, 08 Oct 2019 09:35:48 GMT):
another way, is to use timestamps of course
yacovm (Tue, 08 Oct 2019 09:37:13 GMT):
I think for a permissioned network, this isn't a big issue since it's only a denial of service, and every client that will do that - will also harm its own organization too
yacovm (Tue, 08 Oct 2019 09:38:31 GMT):
In our BFT implementation, you get a resistance against this kind of attack in the same block (you can't put 2 transactions that are the same, in the same block), but not across different blocks.
yacovm (Tue, 08 Oct 2019 09:38:44 GMT):
The trivial way to prevent replay is by epochs or timestamps
guoger (Tue, 08 Oct 2019 09:44:09 GMT):
Thanks for the details! What i was trying to say was, i agree with Jason's statement and we should flip that config, although it doesn't instantly give us `exactly once` semantics from *fabric client's` point of view.
guoger (Tue, 08 Oct 2019 09:44:27 GMT):
and i'd actually like to hear more on epochs in main channel, just not to digress this thread
yacovm (Tue, 08 Oct 2019 09:44:31 GMT):
why not?
guoger (Tue, 08 Oct 2019 09:47:00 GMT):
how does orderer know if a client is resending a tx?
yacovm (Tue, 08 Oct 2019 09:47:22 GMT):
it doesn't need to.
yacovm (Tue, 08 Oct 2019 09:47:36 GMT):
the client doesn't care if the transaction gets 10 times into the ledger
yacovm (Tue, 08 Oct 2019 09:47:49 GMT):
fabric ensures it only treats the first one
guoger (Tue, 08 Oct 2019 09:49:14 GMT):
i think we are talking past each other - basically it does not need, nor provide `exactly once`
yacovm (Tue, 08 Oct 2019 09:50:22 GMT):
well this is semantics... to me, if you as a client send a transaction that transfers your funds to Jason, and you do a re-transmission and both transactions end up in the ledger, but the final result is that you only send money once, then it is exactly once
guoger (Tue, 08 Oct 2019 09:52:28 GMT):
haha, to me that's `most once`, but this is just wording :P
guoger (Tue, 08 Oct 2019 09:52:50 GMT):
i know what you meant
yacovm (Tue, 08 Oct 2019 09:53:23 GMT):
ah, I assumed that we are discussing everything with the assumption that the client got at least one transaction in the orderer
guoger (Tue, 08 Oct 2019 09:55:25 GMT):
@yacovm epochs are signed, right?
yacovm (Tue, 08 Oct 2019 09:55:45 GMT):
everything in the transaction is signed
guoger (Tue, 08 Oct 2019 09:58:51 GMT):
what's the advantage of epoch over seq no? infrequently updated?
guoger (Tue, 08 Oct 2019 09:59:30 GMT):
nvm, there's no way that client can get an up-to-date seq
yacovm (Tue, 08 Oct 2019 10:09:42 GMT):
he can ask a peer...
guoger (Tue, 08 Oct 2019 10:16:11 GMT):
and that can be outdated quickly...
guoger (Tue, 08 Oct 2019 10:17:13 GMT):
unless that seq is not global, but pinned to a session
guoger (Tue, 08 Oct 2019 10:17:53 GMT):
(i'm not talking about fabric config seq, but more of an offset, as in kafka)
jyellick (Tue, 08 Oct 2019 18:56:57 GMT):
We certainly can't eliminate duplicated transactions which a client submits multiple times. But, yes, in this particular instance, I'm just thinking that if we can prevent Kafka duplicating the transactions because of producer timeouts, that would be an improvement. I don't see any obvious downside.
jyellick (Tue, 08 Oct 2019 18:56:57 GMT):
We certainly can't eliminate (with a Kafka setting) duplicated transactions which a client submits multiple times. But, yes, in this particular instance, I'm just thinking that if we can prevent Kafka duplicating the transactions because of producer timeouts, that would be an improvement. I don't see any obvious downside.
jyellick (Tue, 08 Oct 2019 19:00:25 GMT):
It just seemed like an 'obvious setting', that I thought perhaps there was a reason we had opted not to enable it.
jyellick (Tue, 08 Oct 2019 19:00:25 GMT):
It just seemed like such an 'obvious setting', that I thought perhaps there was a reason we had opted not to enable it.
kostas (Tue, 08 Oct 2019 22:40:13 GMT):
Hi everyone, so sorry for the late response. Been a bit busy here. Let me catch up on the thread.
kostas (Tue, 08 Oct 2019 22:40:13 GMT):
Hi everyone, sorry for the late response. Been a bit busy here. Let me catch up on the thread.
kostas (Tue, 08 Oct 2019 22:42:47 GMT):
@yacovm: Do you ever sleep?
yacovm (Tue, 08 Oct 2019 22:44:26 GMT):
Yeah, it's just that my clock has shifted slightly...
kostas (Tue, 08 Oct 2019 22:44:34 GMT):
@jyellick Sorry for the late response.
kostas (Tue, 08 Oct 2019 22:44:41 GMT):
@jyellick Sorry for the late response.
kostas (Tue, 08 Oct 2019 22:44:52 GMT):
> As best as I can tell, when retries are enabled, message duplication is an expected behavior of Kafka.
kostas (Tue, 08 Oct 2019 22:44:53 GMT):
Correct.
kostas (Tue, 08 Oct 2019 22:45:10 GMT):
> However, introduced in v0.11 they allow "exactly once" semantics. I think, turning it on would be as easy as flipping https://github.com/Shopify/sarama/blob/master/config.go#L165-L167 to true.
kostas (Tue, 08 Oct 2019 22:45:16 GMT):
Correct.
kostas (Tue, 08 Oct 2019 22:45:24 GMT):
> I was wondering if you guys ever looked at this setting, and if there's a reason why we left it false?
kostas (Tue, 08 Oct 2019 22:46:37 GMT):
Right. The only reason why we left it to false is because Kafka did not support exactly-once semantics when we started using it.
kostas (Tue, 08 Oct 2019 22:48:12 GMT):
I remember reading the news of them adding exactly-once support back when it happened, but I never got to play around with it. That, and the fact that at the end of the day overproducing won't hurt made me leave that setting as is.
kostas (Tue, 08 Oct 2019 22:48:35 GMT):
I see the argument about retries potentially resulting in a waste space.
kostas (Tue, 08 Oct 2019 22:49:21 GMT):
Before you flip that switch, give me a day or two to check with some folks here at work. We've been bitten by a couple of edge cases with exactly-once turned on for Kafka, and I want to make sure these don't apply for us.
kostas (Tue, 08 Oct 2019 22:50:01 GMT):
I would have caught all of these notifications but apparently RC on my phone logged me out. Argh.
kostas (Tue, 08 Oct 2019 23:06:39 GMT):
> Before you flip that switch, give me a day or two to check with some folks here at work.
kostas (Tue, 08 Oct 2019 23:07:06 GMT):
I haven't parsed it, but apparently it's all covered here: https://kafka-summit.org/sessions/hardening-kafka-replication/
guoger (Wed, 09 Oct 2019 07:53:57 GMT):
IIUC, the fundamental problem in those edge cases presented in the link is that, kafka seems to allow partition with *lagged high watermark* to be elected as leader, even without a quorum...
guoger (Wed, 09 Oct 2019 08:09:56 GMT):
and KAFKA-7128 (as mentioned in video) fixes it by not allowing the partition to join ISR, therefore cannot be elected
guoger (Wed, 09 Oct 2019 08:12:29 GMT):
KIP-320 essentially does the same thing by not allowing partition with lagged epoch to join ISR
guoger (Wed, 09 Oct 2019 08:16:31 GMT):
so i think at least based on the content presented in video, the problem is not tied to `exactly once` semantic. and unfortunately the fixes are released in post-v2 of kafka
guoger (Wed, 09 Oct 2019 08:18:39 GMT):
basically even without flipping the switch, we still suffer from those edge cases... (i'd very like to be wrong)
guoger (Wed, 09 Oct 2019 08:18:53 GMT):
thank @kostas for sharing the video!
guoger (Wed, 09 Oct 2019 08:21:24 GMT):
(i'm actually surprised that kafka has fundamental problem in protocol...)
kostas (Wed, 09 Oct 2019 13:33:20 GMT):
@guoger: Got it - thanks for going through the video and sharing the notes! I plan to do the same during lunch today. Will compare notes, and we can decide on a common course.
kostas (Wed, 09 Oct 2019 16:03:43 GMT):
Confluent requires my email to let me see the video. Alright.
kostas (Wed, 09 Oct 2019 16:03:57 GMT):
Screenshot 2019-10-09 at 08.59.48.png
guoger (Thu, 10 Oct 2019 04:45:57 GMT):
fwiw, here's what's required to flip idempotent switch: https://gerrit.hyperledger.org/r/c/fabric/+/33896
guoger (Thu, 10 Oct 2019 04:46:04 GMT):
cc @jyellick @kostas
kostas (Thu, 10 Oct 2019 06:41:18 GMT):
Went over the video earlier today and you’re right, the issue is not tied to their exactly-once option. So we should be good to go for it.
guoger (Thu, 10 Oct 2019 11:59:47 GMT):
replied in gerrit :)
> Thanks for review. I'd like to see if there's a way to verify this actually solves problem. @jyellick , as this originated from the test around large blocks, are they reproducible?
yacovm (Thu, 10 Oct 2019 12:42:41 GMT):
I'm sure we can just make an integration test with Kafka container that sends big transactions.
It will especially manifest in CI, which is known to run on Raspberry Pie / Arduino
kostas (Mon, 21 Oct 2019 21:46:42 GMT):
https://go-review.googlesource.com/c/go/+/202482
guoger (Tue, 22 Oct 2019 02:07:26 GMT):
looks like the best solution in that context but still looks weird to me :joy:
kostas (Tue, 22 Oct 2019 03:16:17 GMT):
Yeah, I didn't even know you could do that.
guoger (Tue, 22 Oct 2019 03:50:47 GMT):
me neither.. thx for sharing :)
guoger (Tue, 22 Oct 2019 05:29:47 GMT):
looking at https://jira.hyperledger.org/browse/FAB-16887, i'm still seeing `ramledger`, is there an ongoing CR to remove it?
> The orderer used to have the ramledger, the jsonledger, and the fileledger. Both of the former types have been removed, so requiring a type to be specified is confusing and unnecessary.
hawkinggg (Mon, 04 Nov 2019 07:16:28 GMT):
Has joined the channel.
guoger (Wed, 06 Nov 2019 04:23:15 GMT):
@yacovm just wanna confirm with you, we panic on invalid blocks [here](https://github.com/hyperledger/fabric/blob/a5bd17f9ec21241c324f453f0ded3d045bd28ff3/orderer/common/cluster/replication.go#L547-L571). Wouldn't a malicious orderer be able to crash others by sending invalid blocks then? I know it's fine for now since we assume CFT, but I guess this would be revisited later on? thx
jyellick (Wed, 06 Nov 2019 04:25:44 GMT):
@guoger Pretty sure the blocks puller validates the block signature according to the block validation policy prior to returning
jyellick (Wed, 06 Nov 2019 04:26:33 GMT):
(And in a BFT environment, you'd require minimally f+1, if not 2f+1 signatures to be valid)
guoger (Wed, 06 Nov 2019 04:29:53 GMT):
ah, i see `BlockVerifier` now
guoger (Wed, 06 Nov 2019 04:29:54 GMT):
thx
dave.enyeart (Thu, 14 Nov 2019 04:29:10 GMT):
@guoger @jyellick i think https://gerrit.hyperledger.org/r/#/c/fabric/+/34324/ can be merged now for v1.4.4
jyxie2007 (Mon, 18 Nov 2019 04:00:12 GMT):
Has joined the channel.
tengc (Thu, 05 Dec 2019 19:47:48 GMT):
Has joined the channel.
guoger (Fri, 06 Dec 2019 05:24:59 GMT):
do we want to rename etcdraft to raft for 2.0? we now have a mix of them in docs/code. IIUC, the name `etcdraft` was chosen because there might be another impl of Raft in the future. However, i think that's not very likely and other impl could always be named to be different from raft... wdyt @jyellick @kostas
jyellick (Fri, 06 Dec 2019 05:57:06 GMT):
I don't really see it as that confusing, personally, and renaming it would seem to simply add to the confusion.
kostas (Fri, 06 Dec 2019 07:50:27 GMT):
Same thoughts here on keeping things as is, but this is not a strong vote.
guoger (Tue, 17 Dec 2019 02:21:01 GMT):
I'm seeing this flake constantly in azp:
```
------------------------------
• Failure [152.936 seconds]
EndToEnd reconfiguration and onboarding when the orderer certificates are all rotated [It] is still possible to onboard new orderers
/home/vsts/work/1/go/src/github.com/hyperledger/fabric/integration/e2e/etcdraft_reconfig_test.go:310
Timed out after 60.000s.
Expected process to exit. It did not.
/home/vsts/work/1/go/src/github.com/hyperledger/fabric/integration/nwo/deploy.go:111
------------------------------
```
however, that specific test has been completed, that the line that failed is around chaincode instantiation:
```
func InstantiateChaincode(n *Network, channel string, orderer *Orderer, chaincode Chaincode, peer *Peer, checkPeers ...*Peer) {
sess, err := n.PeerAdminSession(peer, commands.ChaincodeInstantiate{
ChannelID: channel,
Orderer: n.OrdererAddress(orderer, ListenPort),
Name: chaincode.Name,
Version: chaincode.Version,
Ctor: chaincode.Ctor,
Policy: chaincode.Policy,
Lang: chaincode.Lang,
CollectionsConfig: chaincode.CollectionsConfig,
})
Expect(err).NotTo(HaveOccurred())
Eventually(sess, n.EventuallyTimeout).Should(gexec.Exit(0))
EnsureInstantiated(n, channel, chaincode.Name, chaincode.Version, checkPeers...)
}
```
not quite sure what's going on...
guoger (Tue, 17 Dec 2019 02:21:01 GMT):
I'm seeing this flake constantly in azp:
```
------------------------------
• Failure [152.936 seconds]
EndToEnd reconfiguration and onboarding when the orderer certificates are all rotated [It] is still possible to onboard new orderers
/home/vsts/work/1/go/src/github.com/hyperledger/fabric/integration/e2e/etcdraft_reconfig_test.go:310
Timed out after 60.000s.
Expected process to exit. It did not.
/home/vsts/work/1/go/src/github.com/hyperledger/fabric/integration/nwo/deploy.go:111
------------------------------
```
however, that specific test has been completed, that the line that failed is around chaincode instantiation:
```
func InstantiateChaincode(n *Network, channel string, orderer *Orderer, chaincode Chaincode, peer *Peer, checkPeers ...*Peer) {
sess, err := n.PeerAdminSession(peer, commands.ChaincodeInstantiate{
ChannelID: channel,
Orderer: n.OrdererAddress(orderer, ListenPort),
Name: chaincode.Name,
Version: chaincode.Version,
Ctor: chaincode.Ctor,
Policy: chaincode.Policy,
Lang: chaincode.Lang,
CollectionsConfig: chaincode.CollectionsConfig,
})
Expect(err).NotTo(HaveOccurred())
Eventually(sess, n.EventuallyTimeout).Should(gexec.Exit(0)) <------ failure line
EnsureInstantiated(n, channel, chaincode.Name, chaincode.Version, checkPeers...)
}
```
not quite sure what's going on...
guoger (Tue, 17 Dec 2019 02:21:01 GMT):
I'm seeing this flake constantly in azp:
```
------------------------------
• Failure [152.936 seconds]
EndToEnd reconfiguration and onboarding when the orderer certificates are all rotated [It] is still possible to onboard new orderers
/home/vsts/work/1/go/src/github.com/hyperledger/fabric/integration/e2e/etcdraft_reconfig_test.go:310
Timed out after 60.000s.
Expected process to exit. It did not.
/home/vsts/work/1/go/src/github.com/hyperledger/fabric/integration/nwo/deploy.go:111
------------------------------
```
however, that specific test has been completed actually, that the line that failed is around chaincode instantiation:
```
func InstantiateChaincode(n *Network, channel string, orderer *Orderer, chaincode Chaincode, peer *Peer, checkPeers ...*Peer) {
sess, err := n.PeerAdminSession(peer, commands.ChaincodeInstantiate{
ChannelID: channel,
Orderer: n.OrdererAddress(orderer, ListenPort),
Name: chaincode.Name,
Version: chaincode.Version,
Ctor: chaincode.Ctor,
Policy: chaincode.Policy,
Lang: chaincode.Lang,
CollectionsConfig: chaincode.CollectionsConfig,
})
Expect(err).NotTo(HaveOccurred())
Eventually(sess, n.EventuallyTimeout).Should(gexec.Exit(0)) <------ failure line
EnsureInstantiated(n, channel, chaincode.Name, chaincode.Version, checkPeers...)
}
```
not quite sure what's going on...
guoger (Tue, 17 Dec 2019 02:21:01 GMT):
I'm seeing this flake constantly in azp:
```
------------------------------
• Failure [152.936 seconds]
EndToEnd reconfiguration and onboarding when the orderer certificates are all rotated [It] is still possible to onboard new orderers
/home/vsts/work/1/go/src/github.com/hyperledger/fabric/integration/e2e/etcdraft_reconfig_test.go:310
Timed out after 60.000s.
Expected process to exit. It did not.
/home/vsts/work/1/go/src/github.com/hyperledger/fabric/integration/nwo/deploy.go:111
------------------------------
```
however, that specific test has been completed actually, the line that failed is around chaincode instantiation:
```
func InstantiateChaincode(n *Network, channel string, orderer *Orderer, chaincode Chaincode, peer *Peer, checkPeers ...*Peer) {
sess, err := n.PeerAdminSession(peer, commands.ChaincodeInstantiate{
ChannelID: channel,
Orderer: n.OrdererAddress(orderer, ListenPort),
Name: chaincode.Name,
Version: chaincode.Version,
Ctor: chaincode.Ctor,
Policy: chaincode.Policy,
Lang: chaincode.Lang,
CollectionsConfig: chaincode.CollectionsConfig,
})
Expect(err).NotTo(HaveOccurred())
Eventually(sess, n.EventuallyTimeout).Should(gexec.Exit(0)) <------ failure line
EnsureInstantiated(n, channel, chaincode.Name, chaincode.Version, checkPeers...)
}
```
not quite sure what's going on...
BrettLogan (Tue, 17 Dec 2019 06:31:03 GMT):
Has joined the channel.
BrettLogan (Tue, 17 Dec 2019 06:31:04 GMT):
^^^ Posted a potential fix in fabric-maintainers
guoger (Fri, 14 Feb 2020 09:39:37 GMT):
@jyellick do you know if anyone has done some memory leak test around 1.4.3 orderer?
guoger (Fri, 14 Feb 2020 09:40:19 GMT):
i recall that our testers once suspected memory leak some time ago?
jyellick (Fri, 14 Feb 2020 18:41:29 GMT):
@guoger Not that I'm aware of, though the 'memory leak' as suspected was definitely a false positive
Abhishekkishor (Thu, 12 Mar 2020 19:47:50 GMT):
Has joined the channel.
conanoc (Tue, 19 May 2020 10:04:03 GMT):
Has left the channel.
dtomczyk (Thu, 11 Jun 2020 15:49:13 GMT):
Has joined the channel.
ShuoWang (Mon, 13 Jul 2020 06:31:47 GMT):
Has joined the channel.
yuki-kon (Fri, 14 Aug 2020 06:35:42 GMT):
Has left the channel.
troyronda (Wed, 28 Oct 2020 17:44:26 GMT):
Has left the channel.
husnain (Tue, 03 Nov 2020 11:44:47 GMT):
Has joined the channel.
cynicalsnail (Wed, 18 Nov 2020 07:01:56 GMT):
Has joined the channel.
Param-S (Thu, 10 Jun 2021 18:15:26 GMT):
Has joined the channel.
jmaric (Mon, 12 Jul 2021 16:38:39 GMT):
Has joined the channel.
s.vahidi (Sun, 23 Jan 2022 12:05:08 GMT):
Has joined the channel.
rjones (Wed, 23 Mar 2022 17:35:23 GMT):
rjones (Wed, 23 Mar 2022 17:35:23 GMT):
rjones (Wed, 23 Mar 2022 17:35:23 GMT):
