tkuhrt (Mon, 05 Jun 2017 22:52:57 GMT):
Discussion surrounding support for Hyperledger Fabric on Kubernetes
tkuhrt (Mon, 05 Jun 2017 22:53:23 GMT):
Hyperledger Fabric on Kubernetes
tkuhrt (Mon, 05 Jun 2017 22:53:25 GMT):
Discussion surrounding support for Hyperledger Fabric on Kubernetes
greg.haskins (Mon, 05 Jun 2017 22:54:42 GMT):
Has joined the channel.
greg.haskins (Mon, 05 Jun 2017 22:54:57 GMT):
@tkuhrt thank you for setting this up
tkuhrt (Mon, 05 Jun 2017 22:55:08 GMT):
np
yacovm (Mon, 05 Jun 2017 22:58:18 GMT):
Has joined the channel.
sarkoi (Mon, 05 Jun 2017 23:15:56 GMT):
Has joined the channel.
jeffgarratt (Mon, 05 Jun 2017 23:16:47 GMT):
Has joined the channel.
cbf (Mon, 05 Jun 2017 23:23:52 GMT):
Has joined the channel.
cbf (Mon, 05 Jun 2017 23:26:05 GMT):
good call
SriramaSharma (Mon, 05 Jun 2017 23:39:56 GMT):
Has joined the channel.
aguchi (Mon, 05 Jun 2017 23:56:17 GMT):
Has joined the channel.
grapebaba (Tue, 06 Jun 2017 00:04:52 GMT):
Has joined the channel.
grapebaba (Tue, 06 Jun 2017 00:05:16 GMT):
:hugging:
grapebaba (Tue, 06 Jun 2017 00:17:03 GMT):
We had a proposal more depend on k8s under internal review, however we might not understand everything, we would like more feedback. https://docs.google.com/document/d/1x4W_hltmIybOkG4JGMX_cpEFcWqoi7aJHIFNHhzed8E
zhipengh (Tue, 06 Jun 2017 00:32:29 GMT):
Has joined the channel.
nnao (Tue, 06 Jun 2017 01:16:09 GMT):
Has joined the channel.
yahtoo (Tue, 06 Jun 2017 02:39:47 GMT):
Has joined the channel.
duwenhui (Tue, 06 Jun 2017 03:33:01 GMT):
Has joined the channel.
tylerdmace (Tue, 06 Jun 2017 04:17:16 GMT):
Has joined the channel.
ssaddem (Tue, 06 Jun 2017 08:47:40 GMT):
Has joined the channel.
tongli (Tue, 06 Jun 2017 13:18:17 GMT):
Has joined the channel.
hmchen (Tue, 06 Jun 2017 13:20:52 GMT):
Has joined the channel.
tongli (Tue, 06 Jun 2017 13:24:13 GMT):
@grapebaba hi, r u around?
grapebaba (Tue, 06 Jun 2017 13:26:39 GMT):
Yeah
tongli (Tue, 06 Jun 2017 13:31:42 GMT):
looked at the doc at the link, saw few yaml file assume they are k8s deployment files. not exactly why TPRs, can you provide a bit more info on that?
grapebaba (Tue, 06 Jun 2017 13:32:31 GMT):
The kind is customized
grapebaba (Tue, 06 Jun 2017 13:32:52 GMT):
So actually there are third party resources
tongli (Tue, 06 Jun 2017 13:33:32 GMT):
@grapebaba I understand it is third party resources, do not understand the reason why you need to do that though.
grapebaba (Tue, 06 Jun 2017 13:34:47 GMT):
I want to customize the spec yaml
tongli (Tue, 06 Jun 2017 13:38:10 GMT):
no other way to do the same thing without creating TPR?
ccorley (Tue, 06 Jun 2017 13:40:12 GMT):
Has joined the channel.
grapebaba (Tue, 06 Jun 2017 13:41:20 GMT):
I don't know the other way.
grapebaba (Tue, 06 Jun 2017 13:47:36 GMT):
BTW, currently i manage the pods self and don't use builtin one such as replicaset, statefulset
grapebaba (Tue, 06 Jun 2017 13:49:23 GMT):
That means we can have more control by ourselves, but obviously it should be more difficult than using built in replicaset, statefulset
tongli (Tue, 06 Jun 2017 14:15:50 GMT):
@grapebaba thanks for your info.
grapebaba (Tue, 06 Jun 2017 14:31:18 GMT):
:relaxed:
matanyahu (Tue, 06 Jun 2017 15:22:53 GMT):
Has joined the channel.
guoger (Tue, 06 Jun 2017 15:25:54 GMT):
Has joined the channel.
catbus (Tue, 06 Jun 2017 15:41:54 GMT):
Has joined the channel.
guruce (Tue, 06 Jun 2017 16:46:17 GMT):
Has joined the channel.
mwagner (Tue, 06 Jun 2017 17:14:43 GMT):
Has joined the channel.
mwagner (Tue, 06 Jun 2017 17:17:31 GMT):
grapebaba - RedhHat is greatly interested in getting Fabric on Kubernetes and OpenShift (not to mention we already have it running on RHEL.)
catbus (Tue, 06 Jun 2017 17:21:47 GMT):
I am interested in running fabric on kubernetes on ubuntu.
phillipl 1 (Tue, 06 Jun 2017 18:12:57 GMT):
Has joined the channel.
Ratnakar (Tue, 06 Jun 2017 19:55:05 GMT):
Has joined the channel.
tongli (Wed, 07 Jun 2017 01:06:27 GMT):
We actually have plans to run fabric on multiple nodes. All the effort can combine I think.
zhipengh (Wed, 07 Jun 2017 01:16:22 GMT):
tongli are you using the operator ?
grapebaba (Wed, 07 Jun 2017 01:30:23 GMT):
@zhipengh i think not:sweat_smile:
grapebaba (Wed, 07 Jun 2017 01:31:38 GMT):
they should have an official internal plan
baohua (Wed, 07 Jun 2017 01:41:22 GMT):
Has joined the channel.
baohua (Wed, 07 Jun 2017 01:43:36 GMT):
the overlay networking is the key to support fabric on multiple nodes, i guess those working for openstack would have good experience.
luckydogchina (Wed, 07 Jun 2017 01:47:41 GMT):
Has joined the channel.
qiang0723 (Wed, 07 Jun 2017 05:54:09 GMT):
Has joined the channel.
tongli (Wed, 07 Jun 2017 12:11:39 GMT):
@zhipengh no, not using operator.
tongli (Wed, 07 Jun 2017 12:11:55 GMT):
@zhipengh thinking using Ansible
greg.haskins (Wed, 07 Jun 2017 12:11:58 GMT):
@DannyWong any more context?
DannyWong (Wed, 07 Jun 2017 12:11:59 GMT):
Has joined the channel.
tongli (Wed, 07 Jun 2017 12:12:32 GMT):
@baohua I have worked on OpenStack for a long time (6 years) and Neutron.
tongli (Wed, 07 Jun 2017 12:12:53 GMT):
@baohua I run two OpenStack clouds within IBM.
tongli (Wed, 07 Jun 2017 12:14:13 GMT):
https://github.com/openstack/interop-workloads/tree/master/workloads/ansible/shade/k8s this is the project I did for OpenStack Design Summit on stage demo
tongli (Wed, 07 Jun 2017 12:14:46 GMT):
that was for OpenStack Design Summit Boston just last month.
DannyWong (Wed, 07 Jun 2017 12:45:45 GMT):
Sorry, gotta resume tmw. Today is anniversary with wife. Will got killed if continue to blockchain
baohua (Wed, 07 Jun 2017 13:02:13 GMT):
@tongli nice! very exciting to have similar-background guys here :) i do believe u will enjoy the new blockchain world~
baohua (Wed, 07 Jun 2017 13:02:13 GMT):
@tongli nice! very excited to have similar-background guys here :) i do believe u will enjoy the new blockchain world~
tongli (Wed, 07 Jun 2017 13:20:57 GMT):
@baohua awesome! thanks.
markparz (Wed, 07 Jun 2017 20:00:18 GMT):
Has joined the channel.
dwakeman (Thu, 08 Jun 2017 23:04:08 GMT):
Has joined the channel.
jrezwan (Sun, 11 Jun 2017 16:14:28 GMT):
Has joined the channel.
DannyWong (Mon, 12 Jun 2017 12:59:16 GMT):
Hi Guys
DannyWong (Mon, 12 Jun 2017 13:00:03 GMT):
last mile from running the Fabric v1 beta on my minikube
DannyWong (Mon, 12 Jun 2017 13:00:06 GMT):
got blocked...
DannyWong (Mon, 12 Jun 2017 13:01:33 GMT):
So my peer deployment.yaml (will change to StatefulSets if everything working)
DannyWong (Mon, 12 Jun 2017 13:01:46 GMT):
```
# CORE_PEER_ADDRESS will be used as the "callback" address in the cmd of cc chain code
# chaincode -peer.address=$CORE_PEER_ADDRESS"
- name: CORE_PEER_ADDRESS
# FIXME peer0.org1 cannot be looked up in the cc container as it is not deployed with K8S (hence, no DNS entry in K8S DNS)
# Cannot use IP (does not matter hard-code IP / downward API to get the POD IP) as the x509 validation will fail
value: peer0.org1:7051
- name: CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE
# Need to use bridge to make sure it is on same network with the PODs
value: bridge
```
DannyWong (Mon, 12 Jun 2017 13:02:41 GMT):
I can deploy orderer, 2orgs peers, create channel, join channel and update anchor peers with the e2e
DannyWong (Mon, 12 Jun 2017 13:03:19 GMT):
and I can create the chaincode docker image as well and able to start it
DannyWong (Mon, 12 Jun 2017 13:04:37 GMT):
but the cc container will die in a few seconds with errors...
```
$ docker logs edf4
2017-06-12 11:02:20.907 UTC [shim] userChaincodeStreamGetter -> ERRO 001 Error trying to connect to local peer: context deadline exceeded
Error starting Simple chaincode: Error trying to connect to local peer: context deadline exceeded
```
DannyWong (Mon, 12 Jun 2017 13:05:46 GMT):
the problem is even they are in the same network
the "startup command" of chaincode container is `chaincode -peer.address=peer0-org1:7051`
DannyWong (Mon, 12 Jun 2017 13:05:58 GMT):
but the chaincode container isn't deployed by K8S
DannyWong (Mon, 12 Jun 2017 13:06:16 GMT):
that's why the peer0-org1 does not exist as a A Record in the K8S DNS
DannyWong (Mon, 12 Jun 2017 13:07:53 GMT):
I have tried to use Downward API to pass the POD IP as well, but failed too
```
$ docker logs a346
2017-06-12 12:36:44.184 UTC [shim] userChaincodeStreamGetter -> ERRO 001 Error trying to connect to local peer: x509: cannot validate certificate for 172.17.0.4 because it doesn't contain any IP SANs
Error starting Simple chaincode: Error trying to connect to local peer: x509: cannot validate certificate for 172.17.0.4 because it doesn't contain any IP SANs
```
DannyWong (Mon, 12 Jun 2017 13:10:15 GMT):
Why Docker Swarm + docker compose is working... is that they are running on same Docker Network (overlay or whatever), which all entries are resolvable with the Swarm internal DNS... ... ...
DannyWong (Mon, 12 Jun 2017 13:17:19 GMT):
I am thinking of making a GitHub for this... it would be nice if people can join me to "Fix" this and make a version of running it on Google CLoud Platform
DannyWong (Mon, 12 Jun 2017 13:17:37 GMT):
Anyone interested?
greg.haskins (Mon, 12 Jun 2017 13:48:18 GMT):
I would suggest to either use a github fork or a gerrit CR to collaborate on the fix if you think its large
greg.haskins (Mon, 12 Jun 2017 13:49:06 GMT):
(and I would suggest sticking with a CR unless you cant, it will be easier to upstream the fix later
DannyWong (Mon, 12 Jun 2017 13:51:25 GMT):
umm... fork from where? It is purely K8S configurations right now.
greg.haskins (Mon, 12 Jun 2017 13:52:02 GMT):
oh, i misunderstood, sorry
greg.haskins (Mon, 12 Jun 2017 13:52:12 GMT):
i thought you were saying we needed a patch to fabric
DannyWong (Mon, 12 Jun 2017 13:52:15 GMT):
all the stuffs I "borrowed" is the e2e script... and the "generateArtefacts", which i customized for K8S
DannyWong (Mon, 12 Jun 2017 13:52:20 GMT):
dun worry...
greg.haskins (Mon, 12 Jun 2017 13:52:27 GMT):
thats fine, i misunderstood, disregard
DannyWong (Mon, 12 Jun 2017 13:52:32 GMT):
no problem!
greg.haskins (Mon, 12 Jun 2017 13:52:52 GMT):
btw: one possibility to explore is the PEER_ADDRESSDETECT thing
greg.haskins (Mon, 12 Jun 2017 13:53:03 GMT):
cant recall the exact name, but will dig
DannyWong (Mon, 12 Jun 2017 13:53:09 GMT):
I know which one
greg.haskins (Mon, 12 Jun 2017 13:53:22 GMT):
ok..issue is probably TLS CN/SAN, im sure
greg.haskins (Mon, 12 Jun 2017 13:53:35 GMT):
but it probably solves the k8s DNS problem
DannyWong (Mon, 12 Jun 2017 13:53:57 GMT):
Message Attachments
DannyWong (Mon, 12 Jun 2017 13:54:10 GMT):
really?
greg.haskins (Mon, 12 Jun 2017 13:54:24 GMT):
thats the one...i think if you set that to true, the peer will surface its discovered IP
greg.haskins (Mon, 12 Jun 2017 13:54:33 GMT):
rather than the name passed in
greg.haskins (Mon, 12 Jun 2017 13:54:45 GMT):
but like I said, that will fix the DNS issue but break TLS
DannyWong (Mon, 12 Jun 2017 13:54:49 GMT):
yes
greg.haskins (Mon, 12 Jun 2017 13:54:49 GMT):
so not likely usable
DannyWong (Mon, 12 Jun 2017 13:55:09 GMT):
I tried before as i used Downward API to pass the POD IP
greg.haskins (Mon, 12 Jun 2017 13:55:16 GMT):
ah, i see
DannyWong (Mon, 12 Jun 2017 13:55:16 GMT):
to the peer.address env
greg.haskins (Mon, 12 Jun 2017 13:55:20 GMT):
sorry, TLDR ;)
DannyWong (Mon, 12 Jun 2017 13:55:22 GMT):
but still DEAD by the TLS thing
greg.haskins (Mon, 12 Jun 2017 13:55:26 GMT):
ill try to catch up on your notes later
DannyWong (Mon, 12 Jun 2017 13:55:29 GMT):
I mentioned earlier as well
DannyWong (Mon, 12 Jun 2017 13:55:36 GMT):
no problem. Thanks mate
DannyWong (Mon, 12 Jun 2017 13:55:55 GMT):
my only feeling is that... IF this is resolved, then the e2e should be running fine!
greg.haskins (Mon, 12 Jun 2017 13:56:31 GMT):
one idea: we could possibly do something with addressAutoDetect=true, TLS off, localhost / k8s-network-policy
greg.haskins (Mon, 12 Jun 2017 13:56:49 GMT):
e.g.hmm
greg.haskins (Mon, 12 Jun 2017 13:56:52 GMT):
nevermind
greg.haskins (Mon, 12 Jun 2017 13:56:57 GMT):
i forgot the peer only surfaces one port
greg.haskins (Mon, 12 Jun 2017 13:57:02 GMT):
(not one specifically for the chaincode)
greg.haskins (Mon, 12 Jun 2017 13:57:27 GMT):
oh well, we'll figure something out
DannyWong (Mon, 12 Jun 2017 13:57:41 GMT):
yes... see if anyone else here can think of something else...
DannyWong (Mon, 12 Jun 2017 13:58:07 GMT):
Thanks @greg.haskins
greg.haskins (Mon, 12 Jun 2017 13:58:38 GMT):
not sure how invasive the change would be, but one idea is to get rid of the notion of IP/TLS for the chaincode connection, at least as an option
greg.haskins (Mon, 12 Jun 2017 13:58:46 GMT):
e.g. use unix-domain-sockets
greg.haskins (Mon, 12 Jun 2017 13:59:17 GMT):
then we could create a pod-wide UDS that is volume mounted to all or something
greg.haskins (Mon, 12 Jun 2017 13:59:44 GMT):
related: separate port for chaincode, only bind to localhost
DannyWong (Mon, 12 Jun 2017 13:59:47 GMT):
but seems not a small change
greg.haskins (Mon, 12 Jun 2017 13:59:52 GMT):
probably not
DannyWong (Mon, 12 Jun 2017 13:59:56 GMT):
for the UDS
DannyWong (Mon, 12 Jun 2017 14:00:29 GMT):
in fact, I think a more appropriate way is to code a SPI to let us config how we want to spin up the chaincode container
greg.haskins (Mon, 12 Jun 2017 14:00:34 GMT):
you'd be surprised, but I agree that separate localhost TCP is probably easier
DannyWong (Mon, 12 Jun 2017 14:00:45 GMT):
one SPI for K8S, one SPI for Docker Swarm
greg.haskins (Mon, 12 Jun 2017 14:00:53 GMT):
what is SPI?
DannyWong (Mon, 12 Jun 2017 14:00:54 GMT):
in fact, just relying on Docker Remote API is not scalable
greg.haskins (Mon, 12 Jun 2017 14:00:55 GMT):
not familar
DannyWong (Mon, 12 Jun 2017 14:01:14 GMT):
Service Provider Interface (reverse of API)
greg.haskins (Mon, 12 Jun 2017 14:01:23 GMT):
ah, ok
DannyWong (Mon, 12 Jun 2017 14:01:35 GMT):
Docker Remote API limits by that machine's memory
greg.haskins (Mon, 12 Jun 2017 14:01:37 GMT):
is that a docker-specific concept, or just speaking generally?
DannyWong (Mon, 12 Jun 2017 14:01:45 GMT):
general speaking
DannyWong (Mon, 12 Jun 2017 14:02:09 GMT):
now, chaincode is a service... not really a contract
greg.haskins (Mon, 12 Jun 2017 14:02:09 GMT):
I def agree that going forward, a more flexible abstraction for the chaincode isolation is warranted
DannyWong (Mon, 12 Jun 2017 14:02:29 GMT):
if we want to have one contract one chaincode, then we end up super many cc containers
DannyWong (Mon, 12 Jun 2017 14:02:46 GMT):
definitely need to create cc container with proper container scheduler
greg.haskins (Mon, 12 Jun 2017 14:02:49 GMT):
so why didnt downward API approach work?
DannyWong (Mon, 12 Jun 2017 14:03:03 GMT):
downward API can just pass the POD_ID?
DannyWong (Mon, 12 Jun 2017 14:03:20 GMT):
```
$ docker logs a346
2017-06-12 12:36:44.184 UTC [shim] userChaincodeStreamGetter -> ERRO 001 Error trying to connect to local peer: x509: cannot validate certificate for 172.17.0.4 because it doesn't contain any IP SANs
Error starting Simple chaincode: Error trying to connect to local peer: x509: cannot validate certificate for 172.17.0.4 because it doesn't contain any IP SANs
```
DannyWong (Mon, 12 Jun 2017 14:03:22 GMT):
then this
greg.haskins (Mon, 12 Jun 2017 14:03:46 GMT):
ok, i might be thinking of a different application of the downward API
greg.haskins (Mon, 12 Jun 2017 14:04:01 GMT):
I was thinking you meant you asked the k8s api to create the container, I think I got you now
DannyWong (Mon, 12 Jun 2017 14:04:19 GMT):
nono
DannyWong (Mon, 12 Jun 2017 14:04:36 GMT):
ur previous understanding is correct
DannyWong (Mon, 12 Jun 2017 14:04:54 GMT):
for long term solution and more scalable solution, i think using K8S api to create cc container
DannyWong (Mon, 12 Jun 2017 14:04:58 GMT):
is quite a must
DannyWong (Mon, 12 Jun 2017 14:05:14 GMT):
just that the Fabric should support different impl (different container scheduler)
greg.haskins (Mon, 12 Jun 2017 14:05:15 GMT):
agreed
greg.haskins (Mon, 12 Jun 2017 14:05:40 GMT):
anyway, a short term hack: we could disable host verification in the shim
DannyWong (Mon, 12 Jun 2017 14:05:53 GMT):
but for the downward API, all I did was try to pass the POD_ID to the peer deployment. YAML
then it fails the TLS validation
greg.haskins (Mon, 12 Jun 2017 14:06:00 GMT):
i need to think about it some more, but that might even be a mid/long term solution because we control the trust root anyway
DannyWong (Mon, 12 Jun 2017 14:06:18 GMT):
Yes, agreed... can try disabling the HOST verification thing
DannyWong (Mon, 12 Jun 2017 14:06:27 GMT):
ahh do u know whether it is configurable?
DannyWong (Mon, 12 Jun 2017 14:06:36 GMT):
or need to change code?
greg.haskins (Mon, 12 Jun 2017 14:06:54 GMT):
so peer::autoAddressDetect=true + shim::hostVerify=false
greg.haskins (Mon, 12 Jun 2017 14:06:57 GMT):
and I think it would work
greg.haskins (Mon, 12 Jun 2017 14:07:11 GMT):
my guess is we have to fix the code
greg.haskins (Mon, 12 Jun 2017 14:07:15 GMT):
but id have to look
DannyWong (Mon, 12 Jun 2017 14:07:21 GMT):
where is the shim config?
DannyWong (Mon, 12 Jun 2017 14:07:36 GMT):
the peer autoaddressdetect I know where it is (core.yaml + env env)
greg.haskins (Mon, 12 Jun 2017 14:07:53 GMT):
looking, but not sure its exposed
greg.haskins (Mon, 12 Jun 2017 14:07:57 GMT):
we might have to patch the code
greg.haskins (Mon, 12 Jun 2017 14:11:02 GMT):
good news, it looks like the shim already has a knob
greg.haskins (Mon, 12 Jun 2017 14:11:13 GMT):
now just seeing if it can be tweaked from the peer without code changes
DannyWong (Mon, 12 Jun 2017 14:11:35 GMT):
ic, let me take a look as well
DannyWong (Mon, 12 Jun 2017 14:12:25 GMT):
Can you point me what file you looking at?
DannyWong (Mon, 12 Jun 2017 14:12:27 GMT):
haha...
DannyWong (Mon, 12 Jun 2017 14:12:30 GMT):
too fast you are :P
greg.haskins (Mon, 12 Jun 2017 14:12:44 GMT):
hold on, i am untangling it
DannyWong (Mon, 12 Jun 2017 14:13:56 GMT):
be right back, wife shouting at me asking me to take bath
DannyWong (Mon, 12 Jun 2017 14:13:58 GMT):
haha
greg.haskins (Mon, 12 Jun 2017 14:15:41 GMT):
I think you can just use the CORE_PEER_TLS_SERVERHOSTOVERRIDE
greg.haskins (Mon, 12 Jun 2017 14:15:53 GMT):
plus autoAddressDetect
greg.haskins (Mon, 12 Jun 2017 14:16:19 GMT):
e.g. set autoAddressDetect=true, and CORE_PEER_TLS_SERVERHOSTOVERRIDE to match your x509
greg.haskins (Mon, 12 Jun 2017 14:16:43 GMT):
that will force the peer to send its dynamic IP, and the shim will disregard the IP and verify against the environment
greg.haskins (Mon, 12 Jun 2017 14:17:02 GMT):
confirming now
greg.haskins (Mon, 12 Jun 2017 14:23:00 GMT):
confirmed, its passed all the way down
greg.haskins (Mon, 12 Jun 2017 14:23:07 GMT):
heres where the peer gives it to the chaincode
greg.haskins (Mon, 12 Jun 2017 14:23:08 GMT):
https://github.com/hyperledger/fabric/blob/v1.0.0-beta/core/chaincode/chaincode_support.go#L363
greg.haskins (Mon, 12 Jun 2017 14:23:40 GMT):
and I also confirmed the shim side
DannyWong (Mon, 12 Jun 2017 14:32:38 GMT):
OK cool!
DannyWong (Mon, 12 Jun 2017 14:39:07 GMT):
So, the CORE_PEER_ADDRESS set as 0.0.0.0:7051?
DannyWong (Mon, 12 Jun 2017 14:39:18 GMT):
as I still gotta specify the port, right?
DannyWong (Mon, 12 Jun 2017 14:46:06 GMT):
Message Attachments
DannyWong (Mon, 12 Jun 2017 14:46:46 GMT):
@greg.haskins u are the man!
greg.haskins (Mon, 12 Jun 2017 14:47:28 GMT):
nice!
mwagner (Mon, 12 Jun 2017 14:53:37 GMT):
@DannyWong @greg.haskins if you have something working I would love to give it a test drive!
greg.haskins (Mon, 12 Jun 2017 15:22:39 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=Z4ss5gi9w94KDjgfB) @DannyWong Im actually not sure whether its needed or defaults to 7051
DannyWong (Mon, 12 Jun 2017 15:22:59 GMT):
Right, I placed it anyway :)
greg.haskins (Mon, 12 Jun 2017 15:30:55 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=Kt55g9CmbmYfyg69w) @DannyWong @muralisr a convenient feature might be something that allows the peer to get the SERVERHOSTOVERRIDE from the x509 itself
muralisr (Mon, 12 Jun 2017 15:30:55 GMT):
Has joined the channel.
greg.haskins (Mon, 12 Jun 2017 15:31:51 GMT):
or perhaps simpler is to just allow the shim to disable checking
muralisr (Mon, 12 Jun 2017 15:36:16 GMT):
@greg.haskins need to read up a bit...
greg.haskins (Mon, 12 Jun 2017 15:36:56 GMT):
@muralisr summary is simple: launching chaincode under kubernetes is a little more stringent than pure docker because of the DNS namespace that the chaincode ends up in
greg.haskins (Mon, 12 Jun 2017 15:37:30 GMT):
e.g. in pure docker, "peer0" and "dev-peer0-mycc-1" end up on the same docker network, and thus have resolvable DNS names
muralisr (Mon, 12 Jun 2017 15:37:31 GMT):
ok.
greg.haskins (Mon, 12 Jun 2017 15:37:44 GMT):
in kubernetes, the peer is launched by k8s, and the chaincode is launched by the peer
greg.haskins (Mon, 12 Jun 2017 15:37:52 GMT):
so they end up namespace isolated from one another
greg.haskins (Mon, 12 Jun 2017 15:38:32 GMT):
a solution we have found is to combine AUTOADDRESSDETECT=true + SERVERHOSTOVERRIDE=$x509::CN
greg.haskins (Mon, 12 Jun 2017 15:39:09 GMT):
and what I was pointing out is that setting SERVERHOSTOVERRIDE=$x509::CN and the x509 is redundant configuration work
greg.haskins (Mon, 12 Jun 2017 15:39:28 GMT):
so, wondering if it can be automated/simplified
muralisr (Mon, 12 Jun 2017 15:39:34 GMT):
because we can pick that up ourselves ?
muralisr (Mon, 12 Jun 2017 15:39:36 GMT):
ok
greg.haskins (Mon, 12 Jun 2017 15:40:13 GMT):
IOW, we _have_ to provide the x509, but why not auto-set the override from it: we already know it
greg.haskins (Mon, 12 Jun 2017 15:40:55 GMT):
honestly though, the most simple and equally correct option might be to simply tell the chaincode to not verify the hostname
greg.haskins (Mon, 12 Jun 2017 15:41:18 GMT):
its not used in the same context in which a typical x509 PKI derives value from
muralisr (Mon, 12 Jun 2017 15:42:16 GMT):
I'm not sure how to disable hostname verification
muralisr (Mon, 12 Jun 2017 15:42:32 GMT):
it might be layered into grpc
muralisr (Mon, 12 Jun 2017 15:44:23 GMT):
also, why do you need AUTOADDRESSDETECT=true ?
muralisr (Mon, 12 Jun 2017 15:44:43 GMT):
oh because you don't know the IP
muralisr (Mon, 12 Jun 2017 15:44:45 GMT):
ok
muralisr (Mon, 12 Jun 2017 15:45:16 GMT):
(ie, that has nothing to do with the TLS)
greg.haskins (Mon, 12 Jun 2017 16:03:28 GMT):
@muralisr I know you can in pure golang sockets, not sure what GRPC surfaces
greg.haskins (Mon, 12 Jun 2017 16:03:31 GMT):
can investigate
muralisr (Mon, 12 Jun 2017 16:04:45 GMT):
I do like the notion of hostname verification though...
greg.haskins (Mon, 12 Jun 2017 17:49:01 GMT):
i am not sure it provides any real security in this context
greg.haskins (Mon, 12 Jun 2017 17:49:19 GMT):
the peer is spoon feeding it all of the parameters
greg.haskins (Mon, 12 Jun 2017 17:49:37 GMT):
(including the cert)
greg.haskins (Mon, 12 Jun 2017 17:49:59 GMT):
I need to think about it a little more, but i am fairly certain it offers no real security benefit in this context
greg.haskins (Mon, 12 Jun 2017 17:50:26 GMT):
the problem in general PKI is that all browsers, etc, have a well known trust root
greg.haskins (Mon, 12 Jun 2017 17:51:26 GMT):
so, I could buy a really cheap verisign cert and then MITM "www.ibm.com" with a hotspot in a coffee shop
greg.haskins (Mon, 12 Jun 2017 17:52:42 GMT):
hostname verification could help thwart that type of attack surface because theres no way verisign should grant me a .crt with CN="www.ibm.com", etc
greg.haskins (Mon, 12 Jun 2017 17:52:53 GMT):
but in this case, the peer is spoonfeeding the trust root anyway
greg.haskins (Mon, 12 Jun 2017 17:53:16 GMT):
(by loading the peer.crt into the chaincode context)
greg.haskins (Mon, 12 Jun 2017 17:53:29 GMT):
@muralisr ^^^
greg.haskins (Mon, 12 Jun 2017 17:54:09 GMT):
so the hostname verification probably has little value
greg.haskins (Mon, 12 Jun 2017 17:54:15 GMT):
i need to think about it more
muralisr (Mon, 12 Jun 2017 17:54:39 GMT):
yeah, just a warm fuzzy :-)
yacovm (Mon, 12 Jun 2017 20:27:32 GMT):
It's easy to disable the hostname override with gRPC
yacovm (Mon, 12 Jun 2017 20:27:32 GMT):
It's easy to disable the hostname override with gRPC @muralisr ^
yacovm (Mon, 12 Jun 2017 20:28:46 GMT):
```
credentials.NewTLS(&tls.Config{
InsecureSkipVerify: true,
})
```
yacovm (Mon, 12 Jun 2017 20:48:50 GMT):
It also disables more stuff however
yacovm (Mon, 12 Jun 2017 20:48:50 GMT):
It also disables more checks however
grapebaba (Tue, 13 Jun 2017 10:23:02 GMT):
It should be improvement abstract cc deploy include more orchestration apis. @muralisr @DannyWong @greg.haskins Can we have a JIRA ticket for tracking this
cbf (Tue, 13 Jun 2017 11:46:39 GMT):
@grapebaba please feel free to create one
naolduga (Wed, 14 Jun 2017 17:17:53 GMT):
Has joined the channel.
Calvin_Heo (Thu, 15 Jun 2017 01:59:26 GMT):
Has joined the channel.
binhn (Fri, 16 Jun 2017 16:23:54 GMT):
Has joined the channel.
sfukazu (Mon, 19 Jun 2017 06:35:11 GMT):
Has joined the channel.
amitkumarj441 (Mon, 19 Jun 2017 08:06:00 GMT):
Has joined the channel.
ashahn (Mon, 19 Jun 2017 13:59:08 GMT):
Has joined the channel.
dongqi (Tue, 20 Jun 2017 08:36:36 GMT):
Has joined the channel.
mraikwar (Tue, 20 Jun 2017 09:03:58 GMT):
Has joined the channel.
SubhraMazumdar (Tue, 20 Jun 2017 09:12:25 GMT):
Has joined the channel.
opiepj (Wed, 21 Jun 2017 00:36:17 GMT):
Has joined the channel.
wajid.poernomo (Wed, 21 Jun 2017 00:38:02 GMT):
Has joined the channel.
wajid.poernomo (Wed, 21 Jun 2017 02:55:43 GMT):
Seen a lot of demos for hyperledger and kubernetes floating around including brooklin/Cloudsoft AMP, is there a "official" hyperledger endorsed project for this stuff?
mraikwar (Wed, 21 Jun 2017 03:28:35 GMT):
is kubernets working fine with fabric v1-beta?
harsha (Wed, 21 Jun 2017 05:03:14 GMT):
Has joined the channel.
DannyWong (Wed, 21 Jun 2017 05:19:10 GMT):
I think so.
DannyWong (Wed, 21 Jun 2017 05:19:21 GMT):
that Cloudsoft AMP is on v0.6
yihuang518 (Wed, 21 Jun 2017 05:31:19 GMT):
Has joined the channel.
jmcnevin (Wed, 21 Jun 2017 14:50:15 GMT):
Has joined the channel.
harsha (Wed, 21 Jun 2017 18:12:29 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=RsKWnCh7oonf6s3Zn) @tongli eliminating SPOF, great idea
DannyWong (Thu, 22 Jun 2017 03:46:27 GMT):
@here , I am starting a discussion thread in #fabric channel in view of event handling (duplicate message handling) for multiple nodes/instances environment.... you guys should be interested as well...
n-someya (Tue, 27 Jun 2017 07:04:26 GMT):
Has joined the channel.
aberfou (Tue, 27 Jun 2017 12:37:08 GMT):
Has joined the channel.
coolsvap (Wed, 28 Jun 2017 07:29:13 GMT):
Has joined the channel.
rezamt (Wed, 28 Jun 2017 08:18:12 GMT):
Has joined the channel.
mariol100 (Wed, 28 Jun 2017 17:18:49 GMT):
Has joined the channel.
jeroiraz (Fri, 30 Jun 2017 14:25:46 GMT):
Has joined the channel.
wajid.poernomo (Sat, 01 Jul 2017 11:56:11 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=4AMFsTMAT8XkKkG7S) @mwagner Did you get any leads on this - looks like Danny is doing something different than the grapebaba
bluecrayon52 (Wed, 05 Jul 2017 12:52:23 GMT):
Has joined the channel.
traviscox (Wed, 05 Jul 2017 13:41:17 GMT):
Has joined the channel.
ersudiplama (Wed, 05 Jul 2017 13:42:55 GMT):
Has joined the channel.
samarcho (Thu, 06 Jul 2017 15:14:03 GMT):
Has joined the channel.
samarcho (Thu, 06 Jul 2017 16:18:53 GMT):
@greg.haskins @DannyWong Hi I have been trying to tailor the docker-compose files at https://github.com/yeasy/docker-compose-files/tree/master/hyperledger/1.0 by converting to a Kubernetes yaml file, but have been unsuccessful so far...can you guys post the yaml that you ran or the instructions to run on Kubernetes?
DannyWong (Thu, 06 Jul 2017 16:20:24 GMT):
basically, CORE_PEER_TLS_SERVERHOSTOVERRIDE and CORE_PEER_ADDRESSAUTODETECT at peer do the trick
DannyWong (Thu, 06 Jul 2017 16:20:47 GMT):
plus having your /host/var/lib/docker socket as volume not PV
samarcho (Thu, 06 Jul 2017 16:36:47 GMT):
@DannyWong which yaml file did you start with? the link that I mentioned above does not even have the CORE_PEER_TLS_SERVERHOSTOVERRIDE variabel and neither is there a semblance of a certificate in the docker-compose.yaml file....I am beginning to think that I need a different starting point
samarcho (Thu, 06 Jul 2017 16:37:20 GMT):
@DannyWong I did convert the docker-compose to the Kubrenetes yaml file
DannyWong (Thu, 06 Jul 2017 16:37:34 GMT):
well, i also started with using kompose to convert the docker compose
DannyWong (Thu, 06 Jul 2017 16:38:02 GMT):
then starting all the jorney to converting it... first kafka (and ZK)
DannyWong (Thu, 06 Jul 2017 16:38:05 GMT):
then orderer
DannyWong (Thu, 06 Jul 2017 16:38:07 GMT):
then peers
DannyWong (Thu, 06 Jul 2017 16:38:11 GMT):
then try to E2E
DannyWong (Thu, 06 Jul 2017 16:38:14 GMT):
take some time...
DannyWong (Thu, 06 Jul 2017 16:38:31 GMT):
Are you trying on K8S cluster or simply minikube
samarcho (Thu, 06 Jul 2017 16:39:08 GMT):
minishift
DannyWong (Thu, 06 Jul 2017 16:39:28 GMT):
then that would be easier
DannyWong (Thu, 06 Jul 2017 16:39:47 GMT):
for the PV, make it host path as per docker volume mount
DannyWong (Thu, 06 Jul 2017 16:40:13 GMT):
except the docker Unix socket
samarcho (Thu, 06 Jul 2017 16:42:12 GMT):
can you point me to which docker-compose file you started with ? There are many and some of them are not even supposed to work
DannyWong (Thu, 06 Jul 2017 16:43:53 GMT):
well, i just start with the one in Fabric github, Maybe u should take the one in e2e_example
DannyWong (Thu, 06 Jul 2017 16:44:02 GMT):
example --> e2e
samarcho (Thu, 06 Jul 2017 16:48:01 GMT):
ok, thanks a lot !
Jay (Mon, 10 Jul 2017 08:39:41 GMT):
Has joined the channel.
gauthampamu (Tue, 11 Jul 2017 21:18:12 GMT):
Has joined the channel.
naohide (Wed, 12 Jul 2017 01:36:24 GMT):
Has joined the channel.
ersudiplama (Wed, 12 Jul 2017 17:28:37 GMT):
Hi, Do anyone have any documentations related to multi host networking between hyper-ledger peers ? if so can someone share that doc? As I tried with docker swarm and other stuff where it throws alot of errors or the peers don't communicate with each other.
Thank you in advance
yacovm (Thu, 13 Jul 2017 07:58:48 GMT):
@ersudiplama do you have to use docker, or can you also use VMs?
ersudiplama (Thu, 13 Jul 2017 13:11:47 GMT):
@yacovm we were trying to look at all the probable options for obtaining the multiple host networking between peers residing at different physical machines or servers. We have achieved the multi host networking different physical machines using docker swarm but we are not sure that as the best option or not. We would like to explore more options so that we can create distributed blockchian network between different peers located at different machines. If anyone have more doc or info about creating distribute network between peers residing at different servers or machine and share that doc with us, we would appreciate that help.
ersudiplama (Thu, 13 Jul 2017 13:11:52 GMT):
Thank you in advance
yacovm (Thu, 13 Jul 2017 13:12:28 GMT):
so, if you have linux VMs and you have SSH keys to all of them you can use https://github.com/yacovm/fabricDeployment
yacovm (Thu, 13 Jul 2017 13:12:40 GMT):
It's just a quick and dirty script that deploys fabric on bare metal VMs
yacovm (Thu, 13 Jul 2017 13:12:47 GMT):
assuming you have password-less SSH access
yacovm (Thu, 13 Jul 2017 13:12:56 GMT):
but it does so for a single org
yacovm (Thu, 13 Jul 2017 13:13:02 GMT):
and a Solo orderer (single instance)
yacovm (Thu, 13 Jul 2017 13:13:08 GMT):
but I guess it can be extended for Kafka too...
yacovm (Thu, 13 Jul 2017 13:13:16 GMT):
I'd suggest to take a look
ersudiplama (Thu, 13 Jul 2017 13:57:10 GMT):
@yacovm We tried to use the docker images of the hyperledger fabric and tried to creating the networking between the peers residing at different physical machine with Linux residing at different network and We faced problem for creating networking between different Orgs and Orderer. As Orderer in one container can find the Org in different container at different machine using the service name specified at compose file ( e followed documentation given at hyperledger fabric to create the docker images ] ) . It is works fine sometime and throw unknown error sometime (for same steps which it worked for ) as I dont know what is happening. We are trying to create blockchain network tow Org infrastructure peers residing at different server. So if anyone have any documentation realted to achivig distributed networking between Orgs( peers) residing at different server or physical machine, it would be great help.
ersudiplama (Thu, 13 Jul 2017 14:03:21 GMT):
@yacovm We tried to use the docker images of the hyperledger fabric using docker container other then fabric on bare mental VM and tried to creating the networking between the peers residing at different physical machine with Linux, located at different network.
ersudiplama (Thu, 13 Jul 2017 14:04:21 GMT):
We faced problem for creating networking between different Orgs and Orderer. As Orderer in one container cannot find the Org in different container at different machine using the service name specified at compose file (We followed the documentation given at hyperledger fabric to use the docker image ). It is works fine sometime with docker swarm and sometime throw unknown error like bad request or grp etc (for same steps which we followed to created the network) as I dont know what is happening.
ersudiplama (Thu, 13 Jul 2017 14:04:59 GMT):
We are trying to create blockchain network between Orderer and two Org infrastructure peers residing at different servers. So if anyone have any documentation related to achieving distributed networking between Orgs( peers) residing at different server or physical machine, it would add great value.
ersudiplama (Thu, 13 Jul 2017 14:05:04 GMT):
Thank you in advance
ankursam (Thu, 13 Jul 2017 14:54:13 GMT):
Has joined the channel.
Moto (Mon, 17 Jul 2017 17:44:16 GMT):
Has joined the channel.
Moto (Mon, 17 Jul 2017 17:45:08 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=cr6XdwG3b57RaqtqF) @ersudiplama did you try using an overlay network?
ersudiplama (Mon, 17 Jul 2017 19:13:46 GMT):
@Moto 10 Ya we tried using overlay network, it works but it is not consistent ( as sometime it throws error for same steps which worked earlier). Do you have any tutorial or documentation of fabric which we can follow?
ersudiplama (Mon, 17 Jul 2017 19:13:46 GMT):
@Moto Ya we tried using overlay network, it works but it is not consistent ( as sometime it throws error for same steps which worked earlier). Do you have any tutorial or documentation of fabric which we can follow?
ersudiplama (Mon, 17 Jul 2017 19:13:51 GMT):
Thank you in advance
tongli (Tue, 18 Jul 2017 13:11:30 GMT):
@ersudiplama @Moto I have been working on that with this project,
tongli (Tue, 18 Jul 2017 13:11:53 GMT):
https://github.com/litong01/fabric-deploy
tongli (Tue, 18 Jul 2017 13:12:44 GMT):
to make fabric working on k8s on multiple nodes, I had to use overlay network (flanneld), this project actually start from basically nothing.
tongli (Tue, 18 Jul 2017 13:14:17 GMT):
it will provision nodes from a cloud (OpenStack, VirtualBox, AWS (in the work now)), then initialize these nodes for them to be ready to run fabric, then setup overlay network, dns, registrator etc. eventually deploy fabric onto multiple nodes with multiple peers, orders, orgs.
tongli (Tue, 18 Jul 2017 13:14:59 GMT):
create genesis block, join all peers to the first channel, install a simple chaincode, instantiate the chaincode, then query.
tongli (Tue, 18 Jul 2017 13:16:03 GMT):
for k8s, the project also setup k8s 1.7.0 env on multiple nodes with k8s dashboard and dns services, then start deploy fabric on.
tongli (Tue, 18 Jul 2017 13:17:05 GMT):
the problem that I have is when I start instantiate chaincode, it feels due to network issues.
tongli (Tue, 18 Jul 2017 13:18:11 GMT):
at this point, I am not really sure if chaincode instantiation assumes that all the parties involved has to share same host or ip or something, I tried various ways with no success, need some expert on chaincode instantiation process to help.
tongli (Tue, 18 Jul 2017 13:18:57 GMT):
also, that project now can provision VirtualBox nodes and starting setting things up on the provisioned the VMs if you have no cloud env.
tongli (Tue, 18 Jul 2017 13:20:24 GMT):
If you just have few physical nodes sitting around, you can also use the project to set up fabric but you will need to make sure that your physical servers have ubuntu 16.04 installed clean and network connection.
Moto (Tue, 18 Jul 2017 13:26:26 GMT):
@tongli yeah I saw your project while I was researching fabric networks yesterday. Ansible looks really interesting.
tongli (Tue, 18 Jul 2017 13:27:25 GMT):
@Moto, it is a tool that is very flexible and can do a lot, for provisioning and installation, it is great.
tongli (Tue, 18 Jul 2017 13:28:04 GMT):
we are working hard to make it THE INSTALLER of the fabric regardless what env you might have.
tongli (Tue, 18 Jul 2017 13:29:00 GMT):
it is working great now on OpenStack cloud (overlay network + docker), having a bit of issues in k8s (currently working on resolving that issue),
tongli (Tue, 18 Jul 2017 13:29:13 GMT):
we are also working on adding provisioning part on AWS as well.
tongli (Tue, 18 Jul 2017 13:29:40 GMT):
I just added virtualbox script, so that you can actually provision few nodes on your virtualbox env and start doing things as well.
ersudiplama (Tue, 18 Jul 2017 13:31:43 GMT):
@tongli Thank you for sharing . If I have any question, I will reach out to you.
tongli (Tue, 18 Jul 2017 14:39:15 GMT):
@ersudiplama yeah, no problem. thanks.
tongli (Tue, 18 Jul 2017 15:18:03 GMT):
@ersudiplama forgot to ask you a question, when you setup multiple peer and orderer in k8s env, do you put peer or orderer container inside a pod and create corresponding services for each peer or orderer?
tongli (Tue, 18 Jul 2017 15:18:03 GMT):
@ersudiplama forgot to ask you a question, when you setup multiple peers and orderers in k8s env, do you put peer or orderer container inside a pod and create corresponding services for each peer or orderer?
tongli (Tue, 18 Jul 2017 15:27:44 GMT):
@here, wonder if anyone else here dealing with k8s doing the similar thing to deploy fabric onto k8s? I assume that bare container on k8s is not the way to go.
ersudiplama (Tue, 18 Jul 2017 16:55:14 GMT):
@tongli Now we are trying to use K8 but not successful so far. In our last iteration, we setup multiple peers and orderer using docker swarm over different physical machine using overlay network.
ersudiplama (Tue, 18 Jul 2017 16:55:53 GMT):
That solution was not consistent
tongli (Tue, 18 Jul 2017 16:57:02 GMT):
@ersudiplama cool, I was successful doing that as well with just docker and overlay network (not swarm),
tongli (Tue, 18 Jul 2017 16:57:21 GMT):
however, when I put everything on k8s, the chaincode instantiation failed.
tongli (Tue, 18 Jul 2017 16:57:53 GMT):
looking at the suggestion that @greg.haskins provided, hope I can figure out a way getting it working on k8s env.
ersudiplama (Tue, 18 Jul 2017 16:59:13 GMT):
@tongli Cool, if it works let us know ... best of luck ...
tongli (Tue, 18 Jul 2017 16:59:39 GMT):
@ersudiplama will report back as soon as I have some results.
greg.haskins (Tue, 18 Jul 2017 17:17:45 GMT):
@tongli I literally have it running right now, so I know its possible
tongli (Tue, 18 Jul 2017 17:18:45 GMT):
@greg.haskins great. can you provide me your deployment yaml file ? are you putting peers and orders inside pods?
greg.haskins (Tue, 18 Jul 2017 17:19:10 GMT):
not easily, statestreet is crazy locked down
greg.haskins (Tue, 18 Jul 2017 17:19:16 GMT):
i can tell you the key elements though
tongli (Tue, 18 Jul 2017 17:19:32 GMT):
oh, that will help as well.
tongli (Tue, 18 Jul 2017 17:20:10 GMT):
let me show you mine deployment files.
tongli (Tue, 18 Jul 2017 17:20:27 GMT):
give me few minutes to run the script. a bit of my network settings.
tongli (Tue, 18 Jul 2017 17:20:39 GMT):
I have k8s running on OpenStack 3 VMs.
tongli (Tue, 18 Jul 2017 17:21:11 GMT):
I setup flanneld overlay network so nodes can talk to each other even when they are in a pod.
greg.haskins (Tue, 18 Jul 2017 17:21:22 GMT):
actually, a colleague wrote the deployment that is currently working, and they did it in a different way than i was expecting
greg.haskins (Tue, 18 Jul 2017 17:21:31 GMT):
will take me a little time to fully decipher what they did
tongli (Tue, 18 Jul 2017 17:21:53 GMT):
that is fine, just need to know the key things.
greg.haskins (Tue, 18 Jul 2017 17:21:55 GMT):
I know we talked about the way Ive done it in the past here on this channel
greg.haskins (Tue, 18 Jul 2017 17:22:01 GMT):
let me see if its still in the history
tongli (Tue, 18 Jul 2017 17:22:06 GMT):
cool.
tongli (Tue, 18 Jul 2017 17:22:16 GMT):
that will help a lot.
greg.haskins (Tue, 18 Jul 2017 17:23:48 GMT):
scroll back to here: https://chat.hyperledger.org/channel/fabric-kubernetes?msg=6FwrYSJKHZgQ8oarC
greg.haskins (Tue, 18 Jul 2017 17:24:13 GMT):
ending here: https://chat.hyperledger.org/channel/fabric-kubernetes?msg=yCEmezpyYswxd8uoo
greg.haskins (Tue, 18 Jul 2017 17:24:58 GMT):
also see the AUTOADDRESS thing
greg.haskins (Tue, 18 Jul 2017 17:25:04 GMT):
thats the other trick
tongli (Tue, 18 Jul 2017 17:25:52 GMT):
need to use both?
greg.haskins (Tue, 18 Jul 2017 17:25:54 GMT):
you basically need the peer to surface its dynamic IP (thats what AUTOADDRESS does) and then tell the chaincode to basically ignore the x509 CN
greg.haskins (Tue, 18 Jul 2017 17:26:01 GMT):
thats what SERVERHOSTOVERRIDE does
greg.haskins (Tue, 18 Jul 2017 17:26:40 GMT):
(and the other part is you need the peer pod to be privelged so it has the rights to drive the docker-api
tongli (Tue, 18 Jul 2017 17:27:09 GMT):
ok, I will make that change.
tongli (Tue, 18 Jul 2017 17:27:18 GMT):
will make these changes and try again.
greg.haskins (Tue, 18 Jul 2017 17:27:23 GMT):
I use openshift variant of k8s though, and my privileged setup via "oc scc" might not be universal
greg.haskins (Tue, 18 Jul 2017 17:27:47 GMT):
but I think there is a pure k8s variant that I will leave as an exercise for the reader ;)
tongli (Tue, 18 Jul 2017 17:28:17 GMT):
@greg.haskins that is totally ok, I setup k8s myself on OpenStack, I can give myself all the rights needed.
greg.haskins (Tue, 18 Jul 2017 17:28:24 GMT):
cool
greg.haskins (Tue, 18 Jul 2017 17:29:01 GMT):
my colleague addressed the problem slightly differently by managing the DNS/docker environment
greg.haskins (Tue, 18 Jul 2017 17:29:03 GMT):
fwiw
tongli (Tue, 18 Jul 2017 17:29:14 GMT):
let me try these things. thanks again for your help. btw, you are putting peers and orderers in pods and make services out of them, right?
greg.haskins (Tue, 18 Jul 2017 17:29:33 GMT):
but since I havent done it that way, I am more comfortable advising on the AUTOADDRESS/HOSTOVERRIDE method
greg.haskins (Tue, 18 Jul 2017 17:29:41 GMT):
that is correct
greg.haskins (Tue, 18 Jul 2017 17:29:57 GMT):
1 peer/orderer per pod
tongli (Tue, 18 Jul 2017 17:30:23 GMT):
right, I did 1 peer inside one pod, then a service per peer pod, or orderer pod.
greg.haskins (Tue, 18 Jul 2017 17:30:35 GMT):
yes, i think thats correct
greg.haskins (Tue, 18 Jul 2017 17:30:45 GMT):
thats how I did it, and I believe how my colleague did it, but I would have to look
greg.haskins (Tue, 18 Jul 2017 17:30:55 GMT):
but thats pretty standard, otherwise you dont get a DNS entry
tongli (Tue, 18 Jul 2017 17:30:56 GMT):
also kafka, zookeeper were done same way.
tongli (Tue, 18 Jul 2017 17:31:13 GMT):
exactly.
tongli (Tue, 18 Jul 2017 17:31:40 GMT):
cool, cool, let me try these tricks, and share my deployment yaml files if they do not work for some reason.
greg.haskins (Tue, 18 Jul 2017 17:31:49 GMT):
the AUTOADDRESS thing _may_ preclude the need at least w.r.t. chaincode, but I think you'd eventually need it to surface the peer to the outside anyway
greg.haskins (Tue, 18 Jul 2017 17:32:06 GMT):
actually for the peers to find each other, too
greg.haskins (Tue, 18 Jul 2017 17:32:12 GMT):
so yeah, its needed...
greg.haskins (Tue, 18 Jul 2017 17:32:12 GMT):
heh
tongli (Tue, 18 Jul 2017 17:32:42 GMT):
so each peer should use that flag or this is only needed for instantiate chaincode?
greg.haskins (Tue, 18 Jul 2017 17:32:54 GMT):
each peer
greg.haskins (Tue, 18 Jul 2017 17:33:14 GMT):
you should set each peer in the env/config to have those settings
tongli (Tue, 18 Jul 2017 17:33:26 GMT):
ah, I do not have that flag for each peer. i c.
greg.haskins (Tue, 18 Jul 2017 17:33:29 GMT):
they will take effect when you try to instantiate chaincode
tongli (Tue, 18 Jul 2017 17:33:45 GMT):
make a lot of sense to me.
tongli (Tue, 18 Jul 2017 17:33:53 GMT):
awesome . thanks again.
greg.haskins (Tue, 18 Jul 2017 17:33:58 GMT):
yvw
greg.haskins (Tue, 18 Jul 2017 17:34:07 GMT):
let us know how you make out
tongli (Tue, 18 Jul 2017 17:41:22 GMT):
@greg.haskins will report back very soon. thanks.
tongli (Tue, 18 Jul 2017 18:14:30 GMT):
@greg.haskins made the peer container all privileged, no luck.
tongli (Tue, 18 Jul 2017 18:15:11 GMT):
will try to add CORE_PEER_TLS_SERVERHOSTOVERRIDE, but do not know what value should it bear, find an example says OBC, which does not make sense to me.
tongli (Wed, 19 Jul 2017 02:13:23 GMT):
@greg.haskins l did use neither CORE_PEER_TLS_SERVERHOSTOVERRIDE nor AUTOADDRESS for peer, what I used was to make sure that these bare chaincode containers created by peers using the dns which point back to the k8s dns services (had to use a nginx to proxy it).
tongli (Wed, 19 Jul 2017 02:14:27 GMT):
@greg.haskins now it is all working in k8s env. Just wanted to report back to you.
greg.haskins (Wed, 19 Jul 2017 02:14:54 GMT):
that sounds like the solution that my colleague is currently using
tongli (Wed, 19 Jul 2017 02:15:35 GMT):
@greg.haskins I will verify few more things tomorrow in terms if privileged container actually makes difference, I have it on.
tongli (Wed, 19 Jul 2017 02:16:11 GMT):
seems to me it is all about networking since the chaincode could not find the peer back, now it can with the dns service help.
greg.haskins (Wed, 19 Jul 2017 02:16:35 GMT):
@tongli the issue will be surrounding the /var/lib/docker.sock
greg.haskins (Wed, 19 Jul 2017 02:16:42 GMT):
and your k8s environment
tongli (Wed, 19 Jul 2017 02:16:56 GMT):
that makes a lot of sense to me.
greg.haskins (Wed, 19 Jul 2017 02:17:27 GMT):
as an example, openshift uses selinux to manage pod privleges and if you just blindly try to volume-map the docker.socket it wont work as the peer pod lacks enough selinux privleges
greg.haskins (Wed, 19 Jul 2017 02:18:23 GMT):
but if you enable elevated privleges in the SCC for the context that the peer runs in, and then set privleged=true, the selinux profile is updated to allow the peer to talk to docker
greg.haskins (Wed, 19 Jul 2017 02:18:25 GMT):
YMMV
tongli (Wed, 19 Jul 2017 02:18:43 GMT):
understood.
tongli (Wed, 19 Jul 2017 02:19:07 GMT):
just happy that it is all working with your helping hands, really appreciate it.
greg.haskins (Wed, 19 Jul 2017 02:19:19 GMT):
yvw, sorry it wasnt quite as I depicted
greg.haskins (Wed, 19 Jul 2017 02:19:39 GMT):
not sure what went wrong with that other approach, but both myself and @DannyWong had success going that route
tongli (Wed, 19 Jul 2017 02:19:41 GMT):
still would like to know what value this env variable should be CORE_PEER_TLS_SERVERHOSTOVERRIDE?
greg.haskins (Wed, 19 Jul 2017 02:19:57 GMT):
IIRC, you want to set that to match your x509 CN
tongli (Wed, 19 Jul 2017 02:20:10 GMT):
oh, i c.
greg.haskins (Wed, 19 Jul 2017 02:20:17 GMT):
CN, or any SANs
greg.haskins (Wed, 19 Jul 2017 02:20:37 GMT):
e.g. peer1.example.com or pee1
greg.haskins (Wed, 19 Jul 2017 02:20:37 GMT):
e.g. peer1.example.com or peer1
tongli (Wed, 19 Jul 2017 02:20:37 GMT):
I made sure hostname and CN match up well.
tongli (Wed, 19 Jul 2017 02:20:56 GMT):
Jason is adding something to allow hostname and CN can be different.
greg.haskins (Wed, 19 Jul 2017 02:21:09 GMT):
this is because if you use the two items together I mentioned, your peer will be advertising something like 172.18.0.23
tongli (Wed, 19 Jul 2017 02:21:19 GMT):
it is to make sure in sitiutations that endpoint and the certificate entry can mismatch.
greg.haskins (Wed, 19 Jul 2017 02:21:23 GMT):
which is unlikely to be in your CN/SAN
greg.haskins (Wed, 19 Jul 2017 02:21:53 GMT):
yes, that is what SERVERHOSTOVERRIDE more or less does, its just in a different context
tongli (Wed, 19 Jul 2017 02:22:06 GMT):
make sense.
greg.haskins (Wed, 19 Jul 2017 02:22:07 GMT):
I think Jason added similar feature for orderer
tongli (Wed, 19 Jul 2017 02:22:36 GMT):
hmmm. I thought it was also for peer. wait, let me pull that patch set out.
greg.haskins (Wed, 19 Jul 2017 02:22:48 GMT):
I might be thinking of a different CR
greg.haskins (Wed, 19 Jul 2017 02:23:12 GMT):
but I saw one recently where he added ORDERER_SERVERHOSTOVERRIDE to pair with PEER_SERVERHOSTOVERRIDE (or something like that)
tongli (Wed, 19 Jul 2017 02:23:15 GMT):
https://gerrit.hyperledger.org/r/#/c/11677/
tongli (Wed, 19 Jul 2017 02:23:23 GMT):
that is the patch, I thought it is all about peer.
greg.haskins (Wed, 19 Jul 2017 02:23:32 GMT):
yeah, thats the one
greg.haskins (Wed, 19 Jul 2017 02:23:46 GMT):
it is a patch for the peer, but it is w.r.t. the connection to the orderer
tongli (Wed, 19 Jul 2017 02:24:03 GMT):
ah. great. great. I am following.
tongli (Wed, 19 Jul 2017 02:24:20 GMT):
did not talk like a fool.
greg.haskins (Wed, 19 Jul 2017 02:24:51 GMT):
I misstated above anyway
greg.haskins (Wed, 19 Jul 2017 02:25:07 GMT):
should have said "similar feature for (the) orderer (connection)"
greg.haskins (Wed, 19 Jul 2017 02:25:16 GMT):
it was confusing how I stated it
tongli (Wed, 19 Jul 2017 02:25:27 GMT):
no issue at all. we are trying to figure out things.
tongli (Wed, 19 Jul 2017 02:25:46 GMT):
here is the project I work on https://github.com/litong01/fabric-deploy
tongli (Wed, 19 Jul 2017 02:26:08 GMT):
now the code fully deploys fabric 1.0.0 onto the OpenStack.
tongli (Wed, 19 Jul 2017 02:26:23 GMT):
we've added provision code (just today) against AWS.
greg.haskins (Wed, 19 Jul 2017 02:26:32 GMT):
cool...ill check it out later, but I have to get up early tomorrow so off to bed for me
tongli (Wed, 19 Jul 2017 02:26:36 GMT):
I will try to deploy onto AWS tomorrow and see what happens.
tongli (Wed, 19 Jul 2017 02:26:48 GMT):
good night @greg.haskins .
greg.haskins (Wed, 19 Jul 2017 02:26:56 GMT):
ciao
htyagi90 (Thu, 20 Jul 2017 17:17:15 GMT):
Has joined the channel.
tongli (Fri, 21 Jul 2017 12:53:17 GMT):
@greg.haskins I have used my fabric-deploy project provisioned on AWS (automatically), and deploy k8s , then fabric, it worked fine.
tongli (Fri, 21 Jul 2017 12:53:48 GMT):
thx.
Emmarock (Wed, 26 Jul 2017 08:49:34 GMT):
Has joined the channel.
Emmarock (Wed, 26 Jul 2017 09:03:33 GMT):
Hello all ... I have a little challenge deploying fabric-peer and membersrvc image to my kubernetes env, I followed this tutorial https://www.xenonstack.com/blog/blockchain-app-deployment-using-microservices-with-kubernetes, I was able to pull the image but it's not running, but I'm presently faced with this issue (standard_init_linux.go:178: exec user process caused "exec format error") I need help on what to do and how to resolve this. thanks
tongli (Wed, 26 Jul 2017 14:50:42 GMT):
@Emmarock what error did you see?
tongli (Wed, 26 Jul 2017 14:51:39 GMT):
@Emmarock what env are you using?
daygee (Thu, 27 Jul 2017 11:13:11 GMT):
Has joined the channel.
daygee (Thu, 27 Jul 2017 12:07:40 GMT):
hi
daygee (Thu, 27 Jul 2017 12:07:48 GMT):
Emmarock and I work together
daygee (Thu, 27 Jul 2017 12:08:05 GMT):
the deployment goes fine on kubernetes
daygee (Thu, 27 Jul 2017 12:08:16 GMT):
but trying to deploy chaincode is the issue
daygee (Thu, 27 Jul 2017 12:08:35 GMT):
Error starting SimpleAsset chaincode: Error trying to connect to local peer: context deadline exceeded
daygee (Thu, 27 Jul 2017 12:08:53 GMT):
command: CORE_CHAINCODE_ID_NAME=mycc CORE_PEER_ADDRESS=10.240.29.7:7051 ./sacc
daygee (Thu, 27 Jul 2017 12:12:02 GMT):
we actually followed steps on this link and made few mods: https://www.xenonstack.com/blog/blockchain-app-deployment-using-microservices-with-kubernetes
rameshthoomu (Thu, 27 Jul 2017 13:00:36 GMT):
Has joined the channel.
daygee (Thu, 27 Jul 2017 13:08:05 GMT):
the kubernetes instance is on a server
cbf (Thu, 27 Jul 2017 14:03:23 GMT):
@daygee I just answered this on SO https://stackoverflow.com/questions/45348191/hyperledger-fabric-v1-0-0-instantiate-chaincode-failed/45352960#45352960
cbf (Thu, 27 Jul 2017 14:03:58 GMT):
but basically, if you scroll back to the discussion by @greg.haskins and @tongli the answer is there
daygee (Thu, 27 Jul 2017 14:40:49 GMT):
hello @cbf
daygee (Thu, 27 Jul 2017 14:41:13 GMT):
I have just gone through the conversation
daygee (Thu, 27 Jul 2017 14:41:33 GMT):
I however don't know what to set CORE_PEER_TLS_SERVERHOSTOVERRIDE to
cbf (Thu, 27 Jul 2017 14:46:20 GMT):
@daygee the peer containers
cbf (Thu, 27 Jul 2017 14:46:20 GMT):
@daygee the peer containers x509 CN https://chat.hyperledger.org/channel/fabric-kubernetes?msg=qKQ8u4Jb3CGjenb87
cbf (Thu, 27 Jul 2017 15:03:10 GMT):
however I am now seeing that there's another approach that worked for @tongli
tongli (Thu, 27 Jul 2017 15:46:51 GMT):
@daygee I do not think that parameter works. at least not with a patch that Jason Yellick put in.
tongli (Thu, 27 Jul 2017 15:47:05 GMT):
With Jason's patch, I think that will work.
tongli (Thu, 27 Jul 2017 15:47:52 GMT):
the point is that the endpoint and the peer id can be different. THe endpoint can be an IP address, the peer id can be something peer1st.orga.
greg.haskins (Thu, 27 Jul 2017 21:01:01 GMT):
@daygee The trick is that SERVERHOSTOVERRIDE must be in the set of x509:{CN, SAN*}
greg.haskins (Thu, 27 Jul 2017 21:02:10 GMT):
All SERVERHOSTOVERRIDE really does it is tells the client-side stack to disregard the hostname/ip that it used to connect to the TLS endpoint, and pretend it used $SERVERHOSTOVERRIDE
greg.haskins (Thu, 27 Jul 2017 21:02:18 GMT):
all the other rules of TLS validation still apply
greg.haskins (Thu, 27 Jul 2017 21:02:58 GMT):
meaning, the TLS server still needs to present an x509 that has $SERVERHOSTOVERRIDE defined either as its CN, SAN::DNS, or SAN::IP
greg.haskins (Thu, 27 Jul 2017 21:03:09 GMT):
I think of it like a DNS alias
greg.haskins (Thu, 27 Jul 2017 21:03:33 GMT):
IOW, setting $SERVERHOSTOVERRIDE is kind of like setting /etc/hosts
jmcnevin (Thu, 27 Jul 2017 21:09:44 GMT):
Running into the following error when attempting to create a channel... any thoughts on where to start with this? `Error: Error connecting due to rpc error: code = Unavailable desc = grpc: the connection is unavailable`
chenshiok (Fri, 28 Jul 2017 01:27:12 GMT):
Has joined the channel.
chenshiok (Fri, 28 Jul 2017 09:11:52 GMT):
Hi,I was testing Hyperledger-fabic v1.0.0 with kubernetes. It says missing image when I instantiate chaincode in the cli,But the image just create successful. I found the answer in this RocketChat. And I set CORE_PEER_ADDRESSAUTODETECT=true and CORE_PEER_TLS_SERVERHOSTOVERRIDE=$x509::CN. But this doesn't work for me. By the way, I disabled TLS in the yaml file for some reason. So CORE_PEER_TLS_SERVERHOSTOVERRIDE may not take effect.
tongli (Fri, 28 Jul 2017 11:29:06 GMT):
@chenshiok , when that happens, I found most of the time due to the chaincode container not being able to find the peer container which creates the chaincode container.
tongli (Fri, 28 Jul 2017 11:31:13 GMT):
@chenshiok not sure about your env, but you probably can check when you manually start a container, and see if you can ping back the container by using the peer id.
chenshiok (Mon, 31 Jul 2017 07:14:55 GMT):
@tongli I start the chaincode container manually, and the container cant't ping other peers. I think the problem maybe the DNS. The chaincode container started by the peer or started manually, the kubernets didn't resolve the peer id.
tongli (Mon, 31 Jul 2017 11:44:35 GMT):
@chenshiok these problems are normally env related. Not sure what kind of env you have, in either swarm like env or k8s env, dns is needed.
tongli (Mon, 31 Jul 2017 11:44:58 GMT):
there are a lot of steps you need to go through to make it working.
tongli (Mon, 31 Jul 2017 12:08:48 GMT):
@baohua hi, baohua, I have submitted a patch set to add all the fabric-deploy code onto cello. Here is the patch set, please take a look when you have time. https://gerrit.hyperledger.org/r/#/c/12059/
baohua (Mon, 31 Jul 2017 12:18:13 GMT):
thanks, tong, fwd to the cello channel, too!
baohua (Mon, 31 Jul 2017 12:18:13 GMT):
thanks, tong, fwded to the cello channel, too!
baohua (Mon, 31 Jul 2017 12:18:13 GMT):
thanks, tong, fwded to the cello channel~
tongli (Mon, 31 Jul 2017 12:26:23 GMT):
let me join that channel.
JanRzepecki (Mon, 31 Jul 2017 14:46:29 GMT):
Has joined the channel.
ChandraLekhaChavva (Mon, 31 Jul 2017 20:43:09 GMT):
Has joined the channel.
ulysses (Tue, 01 Aug 2017 21:20:53 GMT):
Has joined the channel.
MattZee (Wed, 02 Aug 2017 02:22:18 GMT):
Has joined the channel.
greg.haskins (Wed, 02 Aug 2017 13:32:07 GMT):
FYI: https://jira.hyperledger.org/browse/FAB-5578
greg.haskins (Wed, 02 Aug 2017 13:32:21 GMT):
will be updating examples/cluster soon to support generating k8s configurations
cbf (Wed, 02 Aug 2017 18:46:20 GMT):
so, we have a bunch of these
cbf (Wed, 02 Aug 2017 18:46:47 GMT):
not that that is a bad thing, but wouldn't it be nice if we had a canonical version we could share?
cbf (Wed, 02 Aug 2017 18:47:01 GMT):
but that also could be extended as needed
cbf (Wed, 02 Aug 2017 18:48:16 GMT):
and interestingly, it seems to be one of the most requested items on SO and other channels (how do I deploy on multiple hosts, etc)
cbf (Wed, 02 Aug 2017 18:48:31 GMT):
and k8s is definitely an important tagret
cbf (Wed, 02 Aug 2017 18:48:31 GMT):
and k8s is definitely an important target
jdockter (Thu, 03 Aug 2017 02:09:20 GMT):
Has joined the channel.
greg.haskins (Thu, 03 Aug 2017 15:59:38 GMT):
@cbf are there other initiatives to support k8s you can point me at?
greg.haskins (Thu, 03 Aug 2017 16:00:04 GMT):
ideally, we can coordinate here
cbf (Thu, 03 Aug 2017 16:03:30 GMT):
cello, then there's https://gerrit.hyperledger.org/r/c/12059/ which adds ansible orchestration on openstack, virtualbox, aws, azure, etc for docker and k8s and then there is other interest from a rather large supporter of k8s
cbf (Thu, 03 Aug 2017 16:03:50 GMT):
@greg.haskins ^^
greg.haskins (Thu, 03 Aug 2017 16:04:41 GMT):
ty...btw: CR link didnt load for me
cbf (Thu, 03 Aug 2017 18:45:54 GMT):
the number is correct
cbf (Thu, 03 Aug 2017 18:46:20 GMT):
the problem is that the two UIs use different URL schemes - which is inane
cbf (Thu, 03 Aug 2017 18:46:33 GMT):
I'm on the new one
cbf (Thu, 03 Aug 2017 18:46:49 GMT):
@greg.haskins ^^
greg.haskins (Thu, 03 Aug 2017 18:47:28 GMT):
ah, i might simply not have permission then
cbf (Thu, 03 Aug 2017 18:48:25 GMT):
really? that's odd
greg.haskins (Thu, 03 Aug 2017 18:48:26 GMT):
oh, nevermind
greg.haskins (Thu, 03 Aug 2017 18:48:29 GMT):
i get what you mean now
greg.haskins (Thu, 03 Aug 2017 18:48:54 GMT):
I figured you meant there was a new gerrit deployment, and I was getting access denied
greg.haskins (Thu, 03 Aug 2017 18:49:10 GMT):
but you are saying you are using a new view into the same data, but it returns an incompatible URL
cbf (Thu, 03 Aug 2017 18:49:13 GMT):
no, just new and old UI
greg.haskins (Thu, 03 Aug 2017 18:49:25 GMT):
right, i found the CR here https://gerrit.hyperledger.org/r/#/c/12059/
cbf (Thu, 03 Aug 2017 18:49:34 GMT):
but the odd thing is that while that's just a cookie setting, they use different URL path schemes
greg.haskins (Thu, 03 Aug 2017 18:49:38 GMT):
i just misunderstood
greg.haskins (Thu, 03 Aug 2017 18:49:59 GMT):
ABI borkage :rage:
cbf (Thu, 03 Aug 2017 18:50:05 GMT):
indeed
cbf (Thu, 03 Aug 2017 18:50:14 GMT):
thank you Googlers
cbf (Thu, 03 Aug 2017 18:50:17 GMT):
sigh
greg.haskins (Thu, 03 Aug 2017 18:50:46 GMT):
can I turn on the new UI too?
cbf (Thu, 03 Aug 2017 18:50:48 GMT):
seriously, you'd think that a company that makes their $$ from the interwebs would understand how they are to be used
cbf (Thu, 03 Aug 2017 18:50:52 GMT):
yep
greg.haskins (Thu, 03 Aug 2017 18:50:57 GMT):
heh, no kidding
cbf (Thu, 03 Aug 2017 18:51:15 GMT):
too few people really understand REST
cbf (Thu, 03 Aug 2017 18:51:26 GMT):
and most have not read Roy's thesis
greg.haskins (Thu, 03 Aug 2017 18:51:35 GMT):
right
greg.haskins (Thu, 03 Aug 2017 18:52:05 GMT):
i dont see anything obvious in the prefs. how do i get into the club?
cbf (Thu, 03 Aug 2017 18:52:52 GMT):
bottom of page
greg.haskins (Thu, 03 Aug 2017 18:53:10 GMT):
cool, thanks
cbf (Thu, 03 Aug 2017 18:53:22 GMT):
I like it better
cbf (Thu, 03 Aug 2017 18:53:28 GMT):
easier navigation when reviewing
cbf (Thu, 03 Aug 2017 18:53:39 GMT):
takes a little getting used to
cbf (Thu, 03 Aug 2017 18:53:46 GMT):
much faster
greg.haskins (Thu, 03 Aug 2017 18:53:52 GMT):
yeah, foreign so far, but ill give it a chance
greg.haskins (Thu, 03 Aug 2017 18:54:05 GMT):
certainly _looks_ better
greg.haskins (Thu, 03 Aug 2017 18:54:13 GMT):
just need to retrain the muscle memory
cbf (Thu, 03 Aug 2017 18:54:26 GMT):
oh and you need to change your settings so that My Changes is the correct URL :rage:
greg.haskins (Thu, 03 Aug 2017 18:54:58 GMT):
mine seems right out of the can
karumbas (Fri, 04 Aug 2017 11:32:06 GMT):
Has joined the channel.
greg.haskins (Sat, 05 Aug 2017 03:21:14 GMT):
FYI: https://gerrit.hyperledger.org/r/c/12159/
y204990 (Sun, 06 Aug 2017 16:25:28 GMT):
Has joined the channel.
grice_32 (Sun, 06 Aug 2017 19:10:05 GMT):
Has joined the channel.
jane3wang (Mon, 07 Aug 2017 16:31:54 GMT):
Has joined the channel.
sklump (Wed, 09 Aug 2017 13:21:34 GMT):
Has joined the channel.
guce (Thu, 10 Aug 2017 03:48:59 GMT):
Has joined the channel.
rock_martin (Thu, 10 Aug 2017 04:08:12 GMT):
Has joined the channel.
Ambau (Thu, 10 Aug 2017 15:02:06 GMT):
Has joined the channel.
Ambau (Thu, 10 Aug 2017 15:02:26 GMT):
Hello, is there a guide or any other resources on how I can install a basic Fabric instance (as a demo) on Kubernetes?
Moto (Thu, 10 Aug 2017 20:10:23 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=Y3H7ENbvTuGj42jf5) @Ambau li tong has been doing some great work with ansible and kubernetes: https://github.com/litong01/fabric-deploy
qqbxclboy (Fri, 11 Aug 2017 07:45:56 GMT):
Has joined the channel.
machidat731 (Fri, 11 Aug 2017 12:51:08 GMT):
Has joined the channel.
greg.haskins (Fri, 11 Aug 2017 16:22:50 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=Y3H7ENbvTuGj42jf5) @Ambau In addition to @Moto s comment, I also have a CR outstanding to add kubernetes support to fabric.git/examples/cluster
greg.haskins (Fri, 11 Aug 2017 16:22:57 GMT):
https://gerrit.hyperledger.org/r/c/12159
smonfort (Fri, 11 Aug 2017 19:23:09 GMT):
Has joined the channel.
jmcnevin (Tue, 15 Aug 2017 21:12:55 GMT):
I'm attempting to set up TLS communication between peers and orderers, but I'm seeing this error in the peer log.. would anyone have ideas of what I've configured incorrectly? `[peer-1 peer] 2017-08-15 21:09:38.795 UTC [ConnProducer] NewConnection -> ERRO 31c Failed connecting to orderer-2.orderer:7050 , error: x509: certificate signed by unknown authority`
rock_martin (Thu, 17 Aug 2017 05:51:48 GMT):
There needs to be a way to connect a chaincode container to a network. In the case where a user runs a peer on one machine, but wants to run that peers chaincode on a different machine, they may need to have additional control over what network the chaincode container connects to. An example of such a scenario would be wanting the peer to communicate with the chaincode container over a docker overlay network.
NithinPaulCherian (Thu, 17 Aug 2017 13:24:54 GMT):
Has joined the channel.
NithinPaulCherian (Thu, 17 Aug 2017 13:27:00 GMT):
Hi, I used https://ibm-blockchain.github.io/ instructions to setup a Blockchain network on the IBM Container Service's free plan. But it by default connects to leveldb. How can I migrate it to a CouchDB so that I can do complex Queries?
jeffgarratt (Thu, 17 Aug 2017 19:23:00 GMT):
@rock_martin this should be possible today using the CORE_VM_ENDPOINT
jeffgarratt (Thu, 17 Aug 2017 19:23:00 GMT):
@rock_martin this should be possible today using the CORE_VM_ENDPOINT see https://github.com/hyperledger/fabric/blob/release/sampleconfig/core.yaml#L271
idpattison (Fri, 18 Aug 2017 16:48:11 GMT):
Has joined the channel.
zws (Sat, 19 Aug 2017 08:56:05 GMT):
Has joined the channel.
idpattison (Sun, 20 Aug 2017 09:59:26 GMT):
@NithinPaulCherian I'm 90% of the way there to getting HLF with CouchDB working in Kubernetes, perhaps someone can spot the errors here? I've set up one CouchDB service per peer and wired the peer to CouchDB with the CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS variable. The peers connect to CouchDB fine, the databases are created and there is no error message. However when the peers try to join the channel there is an error in the 'joinchannel' pod: `2017-08-20T09:52:34.777594452Z 2017-08-20 09:52:34.777 UTC [grpc] Printf -> DEBU 005 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 10.10.10.62:5010: getsockopt: connection refused"; Reconnecting to {blockchain-org2peer1:5010
idpattison (Sun, 20 Aug 2017 10:01:41 GMT):
Message Attachments
idpattison (Sun, 20 Aug 2017 10:01:58 GMT):
Message Attachments
Luke_Chen (Mon, 21 Aug 2017 05:43:55 GMT):
Has joined the channel.
sklump (Tue, 22 Aug 2017 11:51:03 GMT):
Has left the channel.
DennisM330 (Wed, 23 Aug 2017 05:31:09 GMT):
Has joined the channel.
Hai-XuCheng (Wed, 23 Aug 2017 08:42:53 GMT):
Has joined the channel.
nvlasov (Fri, 25 Aug 2017 07:20:13 GMT):
Has joined the channel.
AuHuR (Fri, 25 Aug 2017 12:06:51 GMT):
Has joined the channel.
rangak (Sat, 26 Aug 2017 19:28:48 GMT):
Has joined the channel.
idpattison (Tue, 29 Aug 2017 09:48:15 GMT):
The Kubernetes config files at https://github.com/IBM-Blockchain/ibm-container-service have been updated to support CouchDB as the database. Use `./create_all.sh --with-couchdb`. Thanks to Mihir Shah for debugging :)
glenlau (Sat, 02 Sep 2017 13:10:09 GMT):
Has joined the channel.
knagware9 (Mon, 04 Sep 2017 11:17:35 GMT):
Has joined the channel.
knagware9 (Mon, 04 Sep 2017 11:19:04 GMT):
Hi ...can you please help me to solve this error during enroll admin on CA ..My cluster is on kubernetes..
knagware9 (Mon, 04 Sep 2017 11:19:38 GMT):
Message Attachments
tylerwince (Mon, 04 Sep 2017 19:14:36 GMT):
Has joined the channel.
knagware9 (Tue, 05 Sep 2017 06:34:57 GMT):
solved this issue...issue with CA service port number
AkshayMisal (Tue, 05 Sep 2017 14:32:20 GMT):
Has joined the channel.
dragon82 (Wed, 06 Sep 2017 08:59:25 GMT):
Has joined the channel.
tiennv (Thu, 07 Sep 2017 02:49:20 GMT):
Has joined the channel.
knagware9 (Thu, 07 Sep 2017 11:40:24 GMT):
Hi ...is there any way to connect bluemix application to Kuberntes cluster ?
cbf (Thu, 14 Sep 2017 15:56:34 GMT):
@knagware9 that Q more relevant on a Bluemix forum
AuHuR (Fri, 15 Sep 2017 08:52:02 GMT):
Has left the channel.
yoyokeen (Tue, 19 Sep 2017 01:26:59 GMT):
Has joined the channel.
mintzhao (Tue, 19 Sep 2017 09:36:58 GMT):
Has joined the channel.
ykcai (Tue, 19 Sep 2017 21:56:09 GMT):
Has joined the channel.
Jacky_Sheng (Wed, 20 Sep 2017 03:21:11 GMT):
Has joined the channel.
knagware9 (Thu, 21 Sep 2017 16:35:24 GMT):
Hi,,
knagware9 (Thu, 21 Sep 2017 16:38:02 GMT):
Upgarded to Kubenetes 1.7.4 on bluemix but hyperledger fabric giving error on setup,It seems docker version issue on Kubernetes.On Kubernetes version 1.5 was working fine.
knagware9 (Thu, 21 Sep 2017 16:40:49 GMT):
Message Attachments
leoleo (Fri, 22 Sep 2017 02:07:42 GMT):
Has joined the channel.
Ryan--Yang (Fri, 22 Sep 2017 03:02:10 GMT):
Has joined the channel.
avi-nyc (Sat, 23 Sep 2017 21:38:54 GMT):
Has joined the channel.
toriaezunama (Mon, 25 Sep 2017 12:17:20 GMT):
Has joined the channel.
cbf (Mon, 25 Sep 2017 12:30:55 GMT):
@knagware9 again, this is more appropriately discussed on the Bluemix forums please
mamtabharadwaj (Tue, 26 Sep 2017 06:16:51 GMT):
Has joined the channel.
mamtabharadwaj (Tue, 26 Sep 2017 06:17:50 GMT):
Message Attachments
yushan (Tue, 26 Sep 2017 09:17:05 GMT):
Has joined the channel.
tkuhrt (Thu, 28 Sep 2017 10:50:20 GMT):
@mamtabharadwaj : Have you asked on the #composer channel?
rock_martin (Thu, 28 Sep 2017 15:06:35 GMT):
Anyone let me know some links for using hyperledger fabric as for production based environment setup on multiple different hosts with respect to docker swarm or kubernates, Thanks in advance
ysim (Thu, 28 Sep 2017 15:35:13 GMT):
Has joined the channel.
joaquimpedrooliveira (Thu, 28 Sep 2017 18:23:44 GMT):
Has joined the channel.
yedendra (Thu, 28 Sep 2017 20:30:28 GMT):
Has joined the channel.
JamesK (Sun, 01 Oct 2017 19:51:12 GMT):
Has joined the channel.
waqasburney (Mon, 02 Oct 2017 06:37:19 GMT):
Has joined the channel.
AnilOner (Mon, 02 Oct 2017 16:26:50 GMT):
Has joined the channel.
mghasletwala (Wed, 04 Oct 2017 06:13:02 GMT):
Has joined the channel.
joaquimpedrooliveira (Wed, 04 Oct 2017 20:26:10 GMT):
Hi, all! I'm trying to adapt the network from fabric-samples/basic-network to run on a local k8s cluster. I can create a channel, join it, install a chaincode, but I'm having trouble during the chaincode instantiation
joaquimpedrooliveira (Wed, 04 Oct 2017 20:26:29 GMT):
From the peer logs, I see that a timeout is ocurring:
joaquimpedrooliveira (Wed, 04 Oct 2017 20:27:52 GMT):
```
2017-10-04 19:43:38.569 UTC [chaincode-platform] generateDockerfile -> DEBU 543
FROM hyperledger/fabric-baseos:x86_64-0.3.2
ADD binpackage.tar /usr/local/bin
LABEL org.hyperledger.fabric.chaincode.id.name="example02" \
org.hyperledger.fabric.chaincode.id.version="1.0" \
org.hyperledger.fabric.chaincode.type="GOLANG" \
org.hyperledger.fabric.version="1.0.2" \
org.hyperledger.fabric.base.version="0.3.2"
ENV CORE_CHAINCODE_BUILDLEVEL=1.0.2
2017-10-04 19:43:38.573 UTC [util] DockerBuild -> DEBU 544 Attempting build with image hyperledger/fabric-ccenv:x86_64-1.0.2
2017-10-04 19:44:07.860 UTC [dockercontroller] deployImage -> DEBU 545 Created image: dev-peer0.blockchain.serpro.gov.br-example02-1.0-07a9f1e5ac7b9e85c34926565c7c3d7d3d465323132ba6a25677ba153d4c8e73
2017-10-04 19:44:07.860 UTC [dockercontroller] Start -> DEBU 546 start-recreated image successfully
2017-10-04 19:44:07.860 UTC [dockercontroller] createContainer -> DEBU 547 Create container: dev-peer0.blockchain.serpro.gov.br-example02-1.0
2017-10-04 19:44:08.076 UTC [dockercontroller] createContainer -> DEBU 548 Created container: dev-peer0.blockchain.serpro.gov.br-example02-1.0-07a9f1e5ac7b9e85c34926565c7c3d7d3d465323132ba6a25677ba153d4c8e73
2017-10-04 19:44:08.573 UTC [dockercontroller] Start -> DEBU 549 Started container dev-peer0.blockchain.serpro.gov.br-example02-1.0
2017-10-04 19:44:08.573 UTC [container] unlockContainer -> DEBU 54a container lock deleted(dev-peer0.blockchain.serpro.gov.br-example02-1.0)
2017-10-04 19:49:08.573 UTC [chaincode] launchAndWaitForRegister -> DEBU 54b stopping due to error while launching Timeout expired while starting chaincode example02:1.0(networkid:dev,peerid:peer0.blockchain.serpro.gov.br,tx:62df707e19b58abbd28b919d3fd311f6f1406d65a0e066830bb21ede30d0252c)
2017-10-04 19:49:08.573 UTC [container] lockContainer -> DEBU 54c waiting for container(dev-peer0.blockchain.serpro.gov.br-example02-1.0) lock
2017-10-04 19:49:08.574 UTC [container] lockContainer -> DEBU 54d got container (dev-peer0.blockchain.serpro.gov.br-example02-1.0) lock
2017-10-04 19:49:08.576 UTC [dockercontroller] stopInternal -> DEBU 54e Stop container dev-peer0.blockchain.serpro.gov.br-example02-1.0(Container not running: dev-peer0.blockchain.serpro.gov.br-example02-1.0)
2017-10-04 19:49:08.578 UTC [dockercontroller] stopInternal -> DEBU 54f Kill container dev-peer0.blockchain.serpro.gov.br-example02-1.0 (API error (500): {"message":"Cannot kill container dev-peer0.blockchain.serpro.gov.br-example02-1.0: Container 7b7b401a1c6b420eec42568ac1dd945ef1d51396632a171fd401d382c38fea30 is not running"}
)
2017-10-04 19:49:08.636 UTC [dockercontroller] stopInternal -> DEBU 550 Removed container dev-peer0.blockchain.serpro.gov.br-example02-1.0
2017-10-04 19:49:08.636 UTC [container] unlockContainer -> DEBU 551 container lock deleted(dev-peer0.blockchain.serpro.gov.br-example02-1.0)
2017-10-04 19:49:08.636 UTC [chaincode] func1 -> DEBU 552 chaincode example02:1.0 launch seq completed
2017-10-04 19:49:08.636 UTC [chaincode] Launch -> ERRO 553 launchAndWaitForRegister failed Timeout expired while starting chaincode example02:1.0(networkid:dev,peerid:peer0.blockchain.serpro.gov.br,tx:62df707e19b58abbd28b919d3fd311f6f1406d65a0e066830bb21ede30d0252c)
2017-10-04 19:49:08.636 UTC [endorser] callChaincode -> DEBU 554 Exit
2017-10-04 19:49:08.636 UTC [endorser] simulateProposal -> ERRO 555 failed to invoke chaincode name:"lscc" on transaction 62df707e19b58abbd28b919d3fd311f6f1406d65a0e066830bb21ede30d0252c, error: Timeout expired while starting chaincode example02:1.0(networkid:dev,peerid:peer0.blockchain.serpro.gov.br,tx:62df707e19b58abbd28b919d3fd311f6f1406d65a0e066830bb21ede30d0252c)
```
joaquimpedrooliveira (Wed, 04 Oct 2017 20:28:22 GMT):
Any tips? (Sorry for the long log messages)
greg.haskins (Thu, 05 Oct 2017 02:47:57 GMT):
@joaquimpedrooliveira its almost certainly related to your network setup
greg.haskins (Thu, 05 Oct 2017 02:48:02 GMT):
two tips
greg.haskins (Thu, 05 Oct 2017 02:48:39 GMT):
1) if you set CORE_VM_DOCKER_ATTACHSTDOUT=true to the peer environment, it will include the output of the chaincode to the peers log
greg.haskins (Thu, 05 Oct 2017 02:49:15 GMT):
this is incredibly helpful to debug this type of thing, but its off by default for defensive postering of the peer (to prevent a malicious/broken chaincode from spamming the peer log
greg.haskins (Thu, 05 Oct 2017 02:49:15 GMT):
this is incredibly helpful to debug this type of thing, but its off by default for defensive posturing of the peer (to prevent a malicious/broken chaincode from spamming the peer log
greg.haskins (Thu, 05 Oct 2017 02:49:52 GMT):
2) My guess is that you arent surfacing a k8s-friendly peer address to the chaincode and it cant find the peer
greg.haskins (Thu, 05 Oct 2017 02:50:20 GMT):
_checks his code to remember how it works_
joaquimpedrooliveira (Thu, 05 Oct 2017 12:41:43 GMT):
@greg.haskins , thank you very much for you help. I'll take a look at these points.
greg.haskins (Thu, 05 Oct 2017 12:43:55 GMT):
@joaquimpedrooliveira i got distracted and forgot to paste the rest of (2)
greg.haskins (Thu, 05 Oct 2017 12:43:59 GMT):
you want
greg.haskins (Thu, 05 Oct 2017 12:44:03 GMT):
``` - name: CORE_PEER_ADDRESSAUTODETECT
value: "true"
- name: CORE_PEER_TLS_SERVERHOSTOVERRIDE
value: peer1```
greg.haskins (Thu, 05 Oct 2017 12:44:16 GMT):
in the peer environment...
greg.haskins (Thu, 05 Oct 2017 12:44:32 GMT):
that will present the right endpoint to the chaincode when its in k9s
greg.haskins (Thu, 05 Oct 2017 12:44:33 GMT):
k8s
greg.haskins (Thu, 05 Oct 2017 12:44:58 GMT):
(SERVERHOSTOVERRIDE should match your x509::[CN/SAN]
joaquimpedrooliveira (Thu, 05 Oct 2017 12:45:07 GMT):
The more I look at the examples, the more environment variables I discover :smile:
greg.haskins (Thu, 05 Oct 2017 12:45:09 GMT):
if you dont use TLS, you can ignore that part
joaquimpedrooliveira (Thu, 05 Oct 2017 12:46:31 GMT):
Is there any docs listing all environment variables and their values and effects?
greg.haskins (Thu, 05 Oct 2017 12:47:07 GMT):
the system uses a library called viper which allows configuration to be specified both as a yaml file as well as environment overrides
greg.haskins (Thu, 05 Oct 2017 12:47:14 GMT):
so the best thing to do is look at the default yaml
greg.haskins (Thu, 05 Oct 2017 12:47:37 GMT):
https://github.com/hyperledger/fabric/blob/release/sampleconfig/core.yaml
greg.haskins (Thu, 05 Oct 2017 12:48:01 GMT):
for instance, if you wanted to set this one: https://github.com/hyperledger/fabric/blob/release/sampleconfig/core.yaml#L38
greg.haskins (Thu, 05 Oct 2017 12:48:26 GMT):
its $FILE_$PATH_$TO_$ITEM
greg.haskins (Thu, 05 Oct 2017 12:48:43 GMT):
so CORE_LOGGING_PEER would be the envvar that corresponds to L38
joaquimpedrooliveira (Thu, 05 Oct 2017 12:49:22 GMT):
Man, that was the tip of the year :D
joaquimpedrooliveira (Thu, 05 Oct 2017 12:50:00 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=PTf8ZEFKgcwwjvx7E) @greg.haskins I found a comment you made here: https://jira.hyperledger.org/browse/FAB-3339?focusedCommentId=28519&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-28519
joaquimpedrooliveira (Thu, 05 Oct 2017 12:50:46 GMT):
And when you said:
>The fact is, the chaincode container doesn't need any services except for the ability to find the peer's TLS endpoint.
I understoond that TLS _must_ be enabled in the peer
joaquimpedrooliveira (Thu, 05 Oct 2017 12:51:01 GMT):
Did I misunderstand what you mean?
greg.haskins (Thu, 05 Oct 2017 12:51:33 GMT):
im not sure what comment you are referring to, but generally speaking you can disable TLS but you generally do it globally
greg.haskins (Thu, 05 Oct 2017 12:51:38 GMT):
e.g. the entire network operates that way
joaquimpedrooliveira (Thu, 05 Oct 2017 12:52:52 GMT):
Ok, so chaincode would work in k8s even if the network is not using TLS?
joaquimpedrooliveira (Thu, 05 Oct 2017 12:53:56 GMT):
I mean, the chaincode container creation and execution
greg.haskins (Thu, 05 Oct 2017 12:54:33 GMT):
yeah, if you turn off TLS in the config, its turned off in the chaincode too
greg.haskins (Thu, 05 Oct 2017 12:55:02 GMT):
https://github.com/hyperledger/fabric/blob/release/sampleconfig/core.yaml#L210
greg.haskins (Thu, 05 Oct 2017 12:55:16 GMT):
or CORE_PEER_TLS_ENABLED=false
joaquimpedrooliveira (Thu, 05 Oct 2017 12:55:24 GMT):
Great. I'll try the configs you mentioned above. Thanks again for your support!
greg.haskins (Thu, 05 Oct 2017 12:55:28 GMT):
not that I recommend you run this way per se
greg.haskins (Thu, 05 Oct 2017 12:55:40 GMT):
but for sanity checking your setup ...
joaquimpedrooliveira (Thu, 05 Oct 2017 12:56:31 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=xqcrzgQ6d66c3jeyi) @greg.haskins That's the idea.
greg.haskins (Thu, 05 Oct 2017 13:01:07 GMT):
@joaquimpedrooliveira i'll just throw this out there too in case its helpful. I have this CR (which I need to update) https://gerrit.hyperledger.org/r/#/c/12159/
greg.haskins (Thu, 05 Oct 2017 13:01:20 GMT):
it adds a "make kubernetes" option to examples/cluster
greg.haskins (Thu, 05 Oct 2017 13:01:40 GMT):
I use a more up to date version of that daily to run fabric on openshift
greg.haskins (Thu, 05 Oct 2017 13:02:09 GMT):
ill clean it up and merge my local fixes to that CR so I can get it merged, but theres still valuable info in there if you are curious how to run on k8s
greg.haskins (Thu, 05 Oct 2017 13:03:31 GMT):
in a nutshell, "make kubernetes" output is a self-contained yaml file that you can then fire in with kubectl create
joaquimpedrooliveira (Thu, 05 Oct 2017 13:03:38 GMT):
I found this issue yesterday: https://jira.hyperledger.org/browse/FAB-3721, that I think is related to your CR
joaquimpedrooliveira (Thu, 05 Oct 2017 13:04:15 GMT):
But I couldn't find the examples/cluster directory in any repo :grimacing:
greg.haskins (Thu, 05 Oct 2017 13:04:46 GMT):
its in fabric
greg.haskins (Thu, 05 Oct 2017 13:05:07 GMT):
https://github.com/hyperledger/fabric/tree/release/examples/cluster
joaquimpedrooliveira (Thu, 05 Oct 2017 13:05:31 GMT):
oops, my bad :D
joaquimpedrooliveira (Thu, 05 Oct 2017 14:19:11 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=zcwdPRM3mbonpBEKH) @greg.haskins It worked! :champagne: :champagne: :champagne:
joaquimpedrooliveira (Thu, 05 Oct 2017 14:19:48 GMT):
I only added the env vars you suggested in the deployment spec. Thank you very much for your help!
joaquimpedrooliveira (Thu, 05 Oct 2017 14:20:13 GMT):
Now I gonna enable TLS and test again :)
joaquimpedrooliveira (Fri, 06 Oct 2017 18:55:22 GMT):
Hi, all. When I enable TLS in my network, I get the following error in *peer* when trying to instantiate a chaincode:
```INFO 55a Error starting Simple chaincode: Error trying to connect to local peer: x509: cannot validate certificate for 192.168.4.33 because it doesn't contain any IP SANs2017-10-06 18:45:59.124 UTC [shim] userChaincodeStreamGetter -> ERRO 002 Error trying to connect to local peer: x509: cannot validate certificate for 192.168.4.33 because it doesn't contain any IP SANs
```
joaquimpedrooliveira (Fri, 06 Oct 2017 18:56:22 GMT):
What certificate is being used here? I cannot remember of any certificate being generated by `cryptogen` for the chaincode container.
joaquimpedrooliveira (Fri, 06 Oct 2017 19:02:08 GMT):
Do I need to configure `chaincodeListenAddress` and expose this port in the deployment spec?
greg.haskins (Fri, 06 Oct 2017 19:36:28 GMT):
@joaquimpedrooliveira this is what the SERVERHOSTOVERRIDE mechanism was designed for
greg.haskins (Fri, 06 Oct 2017 19:36:32 GMT):
heres whats happening:
greg.haskins (Fri, 06 Oct 2017 19:36:50 GMT):
you (or cryptogen, probably) build a set of x509s for your network
greg.haskins (Fri, 06 Oct 2017 19:37:23 GMT):
that x509 has a CommonName and/or Subject Alternative Name that is "x"
greg.haskins (Fri, 06 Oct 2017 19:37:46 GMT):
the chaincode will expect that the URL that it uses should match the x509 presented
greg.haskins (Fri, 06 Oct 2017 19:38:20 GMT):
meaning, the URL that the peer hands to the chaincode is expected to correlated to the CN/SAN that the peer surfaces in its x509
greg.haskins (Fri, 06 Oct 2017 19:38:42 GMT):
the problem on k8s is, everyone has a dynamic address, etc
greg.haskins (Fri, 06 Oct 2017 19:38:52 GMT):
(there are actually deeper problems I wont get into)
greg.haskins (Fri, 06 Oct 2017 19:39:24 GMT):
but long story short, you can work around those k8s-dynamic-environment type problems by using the ADDRESSAUTODETECT feature
greg.haskins (Fri, 06 Oct 2017 19:39:41 GMT):
this means the peer automatically detects its address and gives _that_ to the chaincode
greg.haskins (Fri, 06 Oct 2017 19:39:54 GMT):
the problem is, the dynamic address is almost certainly not in your x509 CN/SAN
greg.haskins (Fri, 06 Oct 2017 19:40:09 GMT):
so, the second part of the equation is to use SERVERHOSTOVERRIDE
greg.haskins (Fri, 06 Oct 2017 19:41:01 GMT):
this means that the chaincode TLS library will use $SERVERHOSTOVERRIDE when validating the x509 CN/SAN rather than the PEER_ADDRESS that was handed to it
greg.haskins (Fri, 06 Oct 2017 19:41:11 GMT):
you can think of SERVERHOSTOVERRIDE as a DNS entry in a way
greg.haskins (Fri, 06 Oct 2017 19:42:07 GMT):
so PEER_ADDRESS = 172.17.0.23 SERVERHOSTOVERRIDE="peer1" means that the chaincode TLS library will connect to grpcs://172.17.0.23 while pretending it connected to grpcs://peer1
greg.haskins (Fri, 06 Oct 2017 19:42:32 GMT):
e.g. as if it had performed a DNS lookup peer1 -> 172.17.0.23
greg.haskins (Fri, 06 Oct 2017 19:43:05 GMT):
SO...the rub is that you want to ensure your SERVERHOSTOVERRIDE is specified when you use k8s, and that it matches your x509
greg.haskins (Fri, 06 Oct 2017 19:43:31 GMT):
hopefully this makes sense
joaquimpedrooliveira (Fri, 06 Oct 2017 20:33:07 GMT):
@greg.haskins , thank you very much for you detailed explanation :)
joaquimpedrooliveira (Fri, 06 Oct 2017 20:33:16 GMT):
you're a life saver!
joaquimpedrooliveira (Fri, 06 Oct 2017 20:35:13 GMT):
for your*
joaquimpedrooliveira (Fri, 06 Oct 2017 20:39:03 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=App6opCvBKYBDBxf6) @greg.haskins I have to configure this property on PEERs deployments, right?
greg.haskins (Fri, 06 Oct 2017 20:59:07 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=xqmcgfwhW5RQ2p8cJ) @joaquimpedrooliveira correct
greg.haskins (Fri, 06 Oct 2017 20:59:40 GMT):
each peer should have ADDRESSAUTODETECT=true and SERVERHOSTOVERRIDE=$peername
joaquimpedrooliveira (Fri, 06 Oct 2017 20:59:58 GMT):
Ok!
greg.haskins (Fri, 06 Oct 2017 20:59:58 GMT):
where $peername is something that correlates to either the x509 CN or one of the SAN::DNS entries
greg.haskins (Fri, 06 Oct 2017 21:00:49 GMT):
fwiw, cryptogen registers both the FQDN as as CN and a relative name as a SAN::DNS
greg.haskins (Fri, 06 Oct 2017 21:01:21 GMT):
e.g. {CN: peer1.org1.com, SAN::DNS: peer1}
m3r00t (Sat, 07 Oct 2017 07:17:59 GMT):
Has joined the channel.
yacovm (Sun, 08 Oct 2017 08:06:16 GMT):
@greg.haskins the above you said made me wonder - why can't we make the peer always create a self-signed CA cert, and then detect its ip address upon boot and generate a TLS certificate with the proper SAN as its IP address and give that + the self signed root CA cert to the chaincode container when it is launched?
yacovm (Sun, 08 Oct 2017 08:06:45 GMT):
(via the upload API of course, not commandline args ;) )
greg.haskins (Sun, 08 Oct 2017 13:10:24 GMT):
@yacovm we could...although its not technically any more secure than SERVERHOSTOVERRIDE as far as I can tell
greg.haskins (Sun, 08 Oct 2017 13:10:56 GMT):
(either way, the chaincode relies on the peer to provide information)
yacovm (Sun, 08 Oct 2017 13:11:27 GMT):
but I thought in your case - the override is supplied not by the peer dynamically but via core.yaml or something?
greg.haskins (Sun, 08 Oct 2017 13:11:47 GMT):
no, the peer sets the override as an envvar
yacovm (Sun, 08 Oct 2017 13:12:00 GMT):
it always does so?
greg.haskins (Sun, 08 Oct 2017 13:12:02 GMT):
but even if it were core.yaml, still doesnt matter
greg.haskins (Sun, 08 Oct 2017 13:12:29 GMT):
i dont recall..it might only set it when its enabled
greg.haskins (Sun, 08 Oct 2017 13:12:29 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=WCqAz8JGfbwc8yhbj) @yacovm i dont recall..it might only set it when its enabled
greg.haskins (Sun, 08 Oct 2017 13:13:11 GMT):
basically, the peer sets envvar, core.yaml, tls material...so I dont think it matters which vehicle is used
greg.haskins (Sun, 08 Oct 2017 13:15:20 GMT):
anyway, i think the SERVERHOSTOVERRIDE is perfectly fine to use..its no different than if we a) hooked the chaincode up properly to the k8s DNS (today its not), or b) dynamically updated /etc/hosts inside the chaincode container, etc
greg.haskins (Sun, 08 Oct 2017 13:15:53 GMT):
its just a convenient way to achieve the same thing as a DNS update
greg.haskins (Sun, 08 Oct 2017 13:16:12 GMT):
and not any less secure
joaquimpedrooliveira (Mon, 09 Oct 2017 14:15:32 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=kJKYBgFGH5bar56Ht) @greg.haskins I set `ADDRESSAUTODETECT=true` and `SERVERHOSTOVERRIDE=$peername`
joaquimpedrooliveira (Mon, 09 Oct 2017 14:16:12 GMT):
I can install the chaincode, but when in try to instantiate it, I get the following error:
```2017-10-09 14:01:36.718 UTC [chaincode] ExecuteChaincode -> ERRO 3e8 Error executing chaincode: Could not get deployment transaction from LSCC for fabcar:1.0 - Get ChaincodeDeploymentSpec for fabcar/serprochannel from LSCC error: chaincode fingerprint mismatch data mismatch
```
greg.haskins (Mon, 09 Oct 2017 14:16:39 GMT):
@joaquimpedrooliveira i've not seen that, but it doesnt appear to be related to the TLS stuff
joaquimpedrooliveira (Mon, 09 Oct 2017 14:17:01 GMT):
I saw two issues in JIRA related to this error message. One of them was corrected in Fabric 1.0.2, that is the version I'm running.
greg.haskins (Mon, 09 Oct 2017 14:17:02 GMT):
@muralisr any ideas?
joaquimpedrooliveira (Mon, 09 Oct 2017 14:17:26 GMT):
The other one remais openned: https://jira.hyperledger.org/browse/FAB-5476
joaquimpedrooliveira (Mon, 09 Oct 2017 14:18:22 GMT):
remains*
muralisr (Mon, 09 Oct 2017 14:19:12 GMT):
@joaquimpedrooliveira let me check something....
muralisr (Mon, 09 Oct 2017 14:19:31 GMT):
(re the "fingerprint mismatch" issue)
joaquimpedrooliveira (Mon, 09 Oct 2017 14:20:22 GMT):
thanks for you help, @muralisr !
muralisr (Mon, 09 Oct 2017 14:21:27 GMT):
of course... can you get me peer logs please ?
joaquimpedrooliveira (Mon, 09 Oct 2017 14:22:08 GMT):
This message was shown in peer log. How many list should I send you?
joaquimpedrooliveira (Mon, 09 Oct 2017 14:22:14 GMT):
How many lines*
joaquimpedrooliveira (Mon, 09 Oct 2017 14:22:18 GMT):
Sorry :)
muralisr (Mon, 09 Oct 2017 14:22:49 GMT):
preferably with CORE_LOGGING_LEVEL=debug env var
muralisr (Mon, 09 Oct 2017 14:23:47 GMT):
hard to say... but the logs for that proposal at least would be good ... would it be possible to "docker logs
muralisr (Mon, 09 Oct 2017 14:23:57 GMT):
and attach peer.log here ?
muralisr (Mon, 09 Oct 2017 14:24:10 GMT):
(unless you don;t want to share sensitiver info)
joaquimpedrooliveira (Mon, 09 Oct 2017 14:24:38 GMT):
I have the following vars set:
`CORE_LOGGING_PEER="debug"` and `CORE_CHAINCODE_LOGGING_LEVEL="DEBUG"`
Is that enough?
joaquimpedrooliveira (Mon, 09 Oct 2017 14:25:13 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=WqpXB2MrYDvLXMnHG) @muralisr I'm running on Kunernetes.
muralisr (Mon, 09 Oct 2017 14:26:07 GMT):
better with CORE_LOGGING_LEVEL="debug" (CORE_LOGGING_PEER="debug" maybe enough
muralisr (Mon, 09 Oct 2017 14:26:32 GMT):
ok. I'm still learning k8s so not sure if that's hard to get in that env
joaquimpedrooliveira (Mon, 09 Oct 2017 14:26:55 GMT):
Right. I'll change my setup here and try again from scratch.
muralisr (Mon, 09 Oct 2017 14:26:59 GMT):
but if you can get the logs for just that proposal that fails, we can start with that
joaquimpedrooliveira (Mon, 09 Oct 2017 14:27:35 GMT):
I'll let you know when I get the logs with the right setup
muralisr (Mon, 09 Oct 2017 14:27:55 GMT):
great.
muralisr (Mon, 09 Oct 2017 14:28:48 GMT):
https://chat.hyperledger.org/channel/fabric-kubernetes?msg=Yq9y7uE9CAmPdtzvW
muralisr (Mon, 09 Oct 2017 14:29:14 GMT):
of course, scratch the ` "docker logs
eodendahl (Mon, 09 Oct 2017 17:54:20 GMT):
Has joined the channel.
ykcai (Mon, 09 Oct 2017 20:43:05 GMT):
Does anyone know why I would be getting this error when running the `create-all script` from IBM containers services
```
2017-10-09 20:41:03.391 UTC [msp] GetDefaultSigningIdentity -> DEBU 002 Obtaining default signing identity
2017-10-09 20:41:03.394 UTC [grpc] Printf -> DEBU 003 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 10.10.10.63:30110: getsockopt: connection refused"; Reconnecting to {blockchain-org1peer1:30110
ykcai (Mon, 09 Oct 2017 20:43:05 GMT):
Does anyone know why I would be getting this error when running the `create-all script` from IBM containers services
```2017-10-09 20:41:03.391 UTC [msp] GetDefaultSigningIdentity -> DEBU 002 Obtaining default signing identity
2017-10-09 20:41:03.394 UTC [grpc] Printf -> DEBU 003 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 10.10.10.63:30110: getsockopt: connection refused"; Reconnecting to {blockchain-org1peer1:30110
ykcai (Mon, 09 Oct 2017 20:43:05 GMT):
Does anyone know why I would be getting this error from `joinchannel` when running the `create-all script` from IBM containers services
```2017-10-09 20:41:03.391 UTC [msp] GetDefaultSigningIdentity -> DEBU 002 Obtaining default signing identity
2017-10-09 20:41:03.394 UTC [grpc] Printf -> DEBU 003 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 10.10.10.63:30110: getsockopt: connection refused"; Reconnecting to {blockchain-org1peer1:30110
joaquimpedrooliveira (Tue, 10 Oct 2017 14:04:44 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=xKABbZM9EzMdvYpyK) @muralisr Hello! Here are the logs that we talked about yesterday:
joaquimpedrooliveira (Tue, 10 Oct 2017 14:05:16 GMT):
When I try to instantiate the chaincode fabcar, from fabric-samples, I noticed an error message:
joaquimpedrooliveira (Tue, 10 Oct 2017 14:06:08 GMT):
Logs from the "cli" container:
```/opt/gopath/src/github.com/hyperledger/fabric/peer# peer chaincode instantiate -o $ORDERER_ADDRESS --tls --cafile $ORDERER_CAFILE -C serprochannel -n fabcar -v 1.0 -c '{"Args":[""]}'
2017-10-10 14:01:36.498 UTC [msp] GetLocalMSP -> DEBU 001 Returning existing local MSP
2017-10-10 14:01:36.498 UTC [msp] GetDefaultSigningIdentity -> DEBU 002 Obtaining default signing identity
2017-10-10 14:01:36.633 UTC [chaincodeCmd] checkChaincodeCmdParams -> INFO 003 Using default escc
2017-10-10 14:01:36.633 UTC [chaincodeCmd] checkChaincodeCmdParams -> INFO 004 Using default vscc
2017-10-10 14:01:36.634 UTC [msp/identity] Sign -> DEBU 005 Sign: plaintext: 0ABB070A6B08031A0C08C0A5F3CE0510...0A000A000A04657363630A0476736363
2017-10-10 14:01:36.634 UTC [msp/identity] Sign -> DEBU 006 Sign: digest: ABBEB6DAD44E0D6FDBF1BBB50683D940F187ABB1EA8C41F302BB72AEE07518C9
Error: Error endorsing chaincode: rpc error: code = Unknown desc = chaincode error (status: 500, message: chaincode exists fabcar)
```
muralisr (Tue, 10 Oct 2017 14:06:54 GMT):
@joaquimpedrooliveira did you attach logs anywhere ?
joaquimpedrooliveira (Tue, 10 Oct 2017 14:07:06 GMT):
Nop. Where can I do it?
joaquimpedrooliveira (Tue, 10 Oct 2017 14:07:13 GMT):
Send here as an attachment?
muralisr (Tue, 10 Oct 2017 14:07:14 GMT):
ok
joaquimpedrooliveira (Tue, 10 Oct 2017 14:07:17 GMT):
pastebin?
muralisr (Tue, 10 Oct 2017 14:07:57 GMT):
I typically rename the file to hae a "mylog.txt" and attach using the clip icon on the side
muralisr (Tue, 10 Oct 2017 14:07:57 GMT):
I typically rename the file to hae a ".txt" as in "mylog.txt" and attach using the clip icon on the side
muralisr (Tue, 10 Oct 2017 14:08:27 GMT):
whatevever works for you though
muralisr (Tue, 10 Oct 2017 14:08:47 GMT):
in the middle of something ... I'll have to look at this later ?
joaquimpedrooliveira (Tue, 10 Oct 2017 14:12:25 GMT):
No problem! :)
joaquimpedrooliveira (Tue, 10 Oct 2017 14:12:37 GMT):
You want the logs from the peer, right?
muralisr (Tue, 10 Oct 2017 14:12:44 GMT):
yes please
joaquimpedrooliveira (Tue, 10 Oct 2017 14:13:07 GMT):
Do the commands I sent from the CLI matter?
joaquimpedrooliveira (Tue, 10 Oct 2017 14:13:57 GMT):
Message Attachments
joaquimpedrooliveira (Tue, 10 Oct 2017 14:14:19 GMT):
Here you are. Thank you very much for your help, @muralisr !
muralisr (Tue, 10 Oct 2017 14:14:31 GMT):
I think a workflow of what you did from the CLI woukd help
joaquimpedrooliveira (Tue, 10 Oct 2017 14:18:41 GMT):
What I did after starting a basic network (one orderer, one peer, one couchdb, one CA):
1. Created channel: `peer channel create -o $ORDERER_ADDRESS --tls --cafile $ORDERER_CAFILE -c serprochannel -f /etc/hyperledger/configtx/channel.tx`
2. Joined channel: `peer channel join --tls --cafile $ORDERER_CAFILE -b serprochannel.block`
3. Installed chaincode `fabcar`: `peer chaincode install --tls --cafile $ORDERER_CAFILE -n fabcar -v 1.0 -p github.com/fabcar`
4. Tried to instantiate the chaincode: `peer chaincode instantiate -o $ORDERER_ADDRESS --tls --cafile $ORDERER_CAFILE -C serprochannel -n fabcar -v 1.0 -c '{"Args":[""]}'`
Here I received the following error: `Error: Error endorsing chaincode: rpc error: code = Unknown desc = chaincode error (status: 500, message: chaincode exists fabcar)`
5. As it says that the chaincode already exists, I tried to invoke it: `peer chaincode invoke -o $ORDERER_ADDRESS --tls --cafile $ORDERER_CAFILE -C serprochannel -n fabcar -c '{"function":"initLedger","Args":[""]}'`
Then I receive: `Error: Error endorsing invoke: rpc error: code = Unknown desc = Error executing chaincode: Could not get deployment transaction from LSCC for fabcar:1.0 - Get ChaincodeDeploymentSpec for fabcar/serprochannel from LSCC error: chaincode fingerprint mismatch data mismatch -
joaquimpedrooliveira (Tue, 10 Oct 2017 14:18:41 GMT):
What I did after starting a basic network (one orderer, one peer, one couchdb, one CA):
1. Created channel: `peer channel create -o $ORDERER_ADDRESS --tls --cafile $ORDERER_CAFILE -c serprochannel -f /etc/hyperledger/configtx/channel.tx`
2. Joined channel: `peer channel join --tls --cafile $ORDERER_CAFILE -b serprochannel.block`
3. Installed chaincode `fabcar`: `peer chaincode install --tls --cafile $ORDERER_CAFILE -n fabcar -v 1.0 -p github.com/fabcar`
4. Tried to instantiate the chaincode: `peer chaincode instantiate -o $ORDERER_ADDRESS --tls --cafile $ORDERER_CAFILE -C serprochannel -n fabcar -v 1.0 -c '{"Args":[""]}'`
Here I received the following error: `Error: Error endorsing chaincode: rpc error: code = Unknown desc = chaincode error (status: 500, message: chaincode exists fabcar)`
5. As it says that the chaincode already exists, I tried to invoke it: `peer chaincode invoke -o $ORDERER_ADDRESS --tls --cafile $ORDERER_CAFILE -C serprochannel -n fabcar -c '{"function":"initLedger","Args":[""]}'`
Then I receive: `Error: Error endorsing invoke: rpc error: code = Unknown desc = Error executing chaincode: Could not get deployment transaction from LSCC for fabcar:1.0 - Get ChaincodeDeploymentSpec for fabcar/serprochannel from LSCC error: chaincode fingerprint mismatch data mismatch -
joaquimpedrooliveira (Tue, 10 Oct 2017 14:19:31 GMT):
Running the default hyperledger-fabric-* containers on Kubernetes
joaquimpedrooliveira (Tue, 10 Oct 2017 14:21:31 GMT):
I have the logs for each individual operation, if it helps.
muralisr (Tue, 10 Oct 2017 14:33:38 GMT):
@joaquimpedrooliveira my guess is this ... it was instantiated on the channel but was not on this particular peer you were trying to invoke
muralisr (Tue, 10 Oct 2017 14:34:07 GMT):
to make that happen, you (rightly) installed the chaincode on the peer (step 3)
muralisr (Tue, 10 Oct 2017 14:34:48 GMT):
however the invoke tried to match the installed CC with the fingerprint of the instnatiated CC for the channel
muralisr (Tue, 10 Oct 2017 14:34:48 GMT):
however the invoke tried to match the installed CC with the fingerprint of the instnatiated CC for the channel before invoking it
muralisr (Tue, 10 Oct 2017 14:36:35 GMT):
and that did not match for some reason ... this could happen if the second install in 3 was different from the chaincode that was used to install in some manner
joaquimpedrooliveira (Tue, 10 Oct 2017 14:40:28 GMT):
@muralisr , I started the network from scratch, so I'm assuming that there was no chaincode previously installed anywhere . Is my assumption ok?
muralisr (Tue, 10 Oct 2017 14:42:06 GMT):
@joaquimpedrooliveira the chaincode did exist on the ledger in 4 ... I'd have expected it to go through successfully if starting from scratch
joaquimpedrooliveira (Tue, 10 Oct 2017 14:43:19 GMT):
it exists in 4 because it was installed in step 3, right?
joaquimpedrooliveira (Tue, 10 Oct 2017 14:43:57 GMT):
I don't understand when you say that
> it was instantiated on the channel but was not on this particular peer you were trying to invoke
joaquimpedrooliveira (Tue, 10 Oct 2017 14:44:59 GMT):
How can I instantiate a chaincode in channel, but not on a peer? I thought that the command `peer chaincode install` did both :)
joaquimpedrooliveira (Tue, 10 Oct 2017 14:45:32 GMT):
I confess I really don't have a deep understand of chaincode lifecycle.
joaquimpedrooliveira (Tue, 10 Oct 2017 14:46:17 GMT):
And a curious thing is: when I did the same steps before, with TLS disabled, it worked.
joaquimpedrooliveira (Tue, 10 Oct 2017 14:47:17 GMT):
Just for clarification: I did the steps above only once each
joaquimpedrooliveira (Tue, 10 Oct 2017 14:47:28 GMT):
I have only one peer in my simple network
muralisr (Tue, 10 Oct 2017 14:47:31 GMT):
step 3 is "install" -> put the chaincode on the peer , step 4 is "instantiate" -> use the installed chaincode to instantiate the chaincode on the channel's ledger
joaquimpedrooliveira (Tue, 10 Oct 2017 14:48:02 GMT):
Now I got it. Thanks for the explanation :)
muralisr (Tue, 10 Oct 2017 14:48:18 GMT):
sure
muralisr (Tue, 10 Oct 2017 14:50:07 GMT):
4 is the key...for some reason the channel is still there and the chaincode is alreay instantiated though you want it to start from scratch
joaquimpedrooliveira (Tue, 10 Oct 2017 14:50:20 GMT):
The couchdb only maintains the world state, not the full ledger, right?
joaquimpedrooliveira (Tue, 10 Oct 2017 14:50:40 GMT):
I'm not restarting it. Could this be the cause?
muralisr (Tue, 10 Oct 2017 14:52:41 GMT):
couchdb maintains everithing ... in particular it would know about instatiated chaincodes
muralisr (Tue, 10 Oct 2017 14:53:32 GMT):
if the clean up did not include removing the ledger dbs, then on restart you'll find it
joaquimpedrooliveira (Tue, 10 Oct 2017 14:53:48 GMT):
So a previous chaincode may exist in couchdb
muralisr (Tue, 10 Oct 2017 14:53:52 GMT):
need to get back... talk later
muralisr (Tue, 10 Oct 2017 14:53:53 GMT):
rigfht
joaquimpedrooliveira (Tue, 10 Oct 2017 14:53:53 GMT):
It may be the cause
muralisr (Tue, 10 Oct 2017 14:53:55 GMT):
right
joaquimpedrooliveira (Tue, 10 Oct 2017 14:54:02 GMT):
Ok, thank you!
joaquimpedrooliveira (Tue, 10 Oct 2017 14:54:13 GMT):
Time to pick up kids at school :)
jmcnevin (Tue, 10 Oct 2017 19:14:23 GMT):
i'm running my peers as a stateful set, and I'm not sure if I should be doing something in particular, but I'm having a difficult time deploying some composer BNAs to my cluster:
```
2017-10-10 19:13:05.497 UTC [dev-peer-0.peer-lynnhurst-composer-0.13.2] func2 -> INFO 44b 2017-10-10 19:13:05.496 UTC [Composer] Info -> INFO 001 Setting the Composer pool size to 8
2017-10-10 19:13:08.500 UTC [dev-peer-0.peer-lynnhurst-composer-0.13.2] func2 -> INFO 44c Error starting chaincode: Error trying to connect to local peer: context deadline exceeded2017-10-10 19:13:08.498 UTC [shim] userChaincodeStreamGetter -> ERRO 002 Error trying to connect to local peer: context deadline exceeded
2017-10-10 19:13:08.530 UTC [dockercontroller] func2 -> INFO 44d Container dev-peer-0.peer-lynnhurst-composer-0.13.2 has closed its IO channel
```
yoheiueda (Wed, 11 Oct 2017 05:28:22 GMT):
Has joined the channel.
joaquimpedrooliveira (Wed, 11 Oct 2017 13:10:26 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=EpHKAYvZrNrAEaKbA) @muralisr I tried again all steps using a clean couchdb and the error changed:
```2017-10-11 12:55:34.038 UTC [dev-peer0.blockchain.serpro.gov.br-fabcar-1.0] func2 -> INFO 396 2017-10-11 12:55:34.038 UTC [shim] userChaincodeStreamGetter -> ERRO 002 Error trying to connect to local peer: x509: certificate signed by unknown authority
2017-10-11 12:55:34.110 UTC [dockercontroller] func2 -> INFO 397 Container dev-peer0.blockchain.serpro.gov.br-fabcar-1.0 has closed its IO channel
2017-10-11 13:00:34.024 UTC [chaincode] launchAndWaitForRegister -> DEBU 398 stopping due to error while launching Timeout expired while starting chaincode fabcar:1.0(networkid:dev,peerid:peer0.blockchain.serpro.gov.br,tx:804d482bb3820b40d27580b464e0bf4434589acd7ed8733cf163a515a97cecdc)
```
joaquimpedrooliveira (Wed, 11 Oct 2017 13:10:43 GMT):
I generated all certificates using `cryptogen`
joaquimpedrooliveira (Wed, 11 Oct 2017 13:10:52 GMT):
The log above is from the peer node
muralisr (Wed, 11 Oct 2017 13:14:01 GMT):
@joaquimpedrooliveira that sounds like a MSP setup error .... there should be some error on the CLI as well
joaquimpedrooliveira (Wed, 11 Oct 2017 13:14:31 GMT):
cli got a timeout:
```/opt/gopath/src/github.com/hyperledger/fabric/peer# peer chaincode instantiate -o $ORDERER_ADDRESS --tls --cafile $ORDERER_CAFILE -C serprochannel -n fabcar -v 1.0 -c '{"Args":[""]}'
2017-10-11 13:02:29.187 UTC [msp] GetLocalMSP -> DEBU 001 Returning existing local MSP
2017-10-11 13:02:29.187 UTC [msp] GetDefaultSigningIdentity -> DEBU 002 Obtaining default signing identity
2017-10-11 13:02:29.320 UTC [chaincodeCmd] checkChaincodeCmdParams -> INFO 003 Using default escc
2017-10-11 13:02:29.321 UTC [chaincodeCmd] checkChaincodeCmdParams -> INFO 004 Using default vscc
2017-10-11 13:02:29.321 UTC [msp/identity] Sign -> DEBU 005 Sign: plaintext: 0ABB070A6B08031A0C08E5ACF8CE0510...0A000A000A04657363630A0476736363
2017-10-11 13:02:29.321 UTC [msp/identity] Sign -> DEBU 006 Sign: digest: C1928AD38710CA4BC81447E7252F2666A57A3E35E89284F3F6FC7EDE2BE581F5
Error: Error endorsing chaincode: rpc error: code = Unknown desc = Timeout expired while starting chaincode fabcar:1.0(networkid:dev,peerid:peer0.blockchain.serpro.gov.br,tx:804d482bb3820b40d27580b464e0bf4434589acd7ed8733cf163a515a97cecdc)
`
joaquimpedrooliveira (Wed, 11 Oct 2017 13:19:24 GMT):
The message below:
```2017-10-11 12:55:34.038 UTC [dev-peer0.blockchain.serpro.gov.br-fabcar-1.0] func2 -> INFO 396 2017-10-11 12:55:34.038 UTC [shim] userChaincodeStreamGetter -> ERRO 002 Error trying to connect to local peer: x509: certificate signed by unknown authority
was sent by the chaincode container while trying to connect to the peer node that instantiated it?
joaquimpedrooliveira (Wed, 11 Oct 2017 13:19:24 GMT):
The message below:
```2017-10-11 12:55:34.038 UTC [dev-peer0.blockchain.serpro.gov.br-fabcar-1.0] func2 -> INFO 396 2017-10-11 12:55:34.038 UTC [shim] userChaincodeStreamGetter -> ERRO 002 Error trying to connect to local peer: x509: certificate signed by unknown authority
```
was sent by the chaincode container while trying to connect to the peer node that instantiated it?
greg.haskins (Wed, 11 Oct 2017 13:20:40 GMT):
@joaquimpedrooliveira just jumping in here, but you might be hitting a known bug
joaquimpedrooliveira (Wed, 11 Oct 2017 13:21:05 GMT):
How lucky I am! :smile:
greg.haskins (Wed, 11 Oct 2017 13:21:13 GMT):
there is a workaround if so
joaquimpedrooliveira (Wed, 11 Oct 2017 13:21:14 GMT):
Which one, @greg.haskins ?
greg.haskins (Wed, 11 Oct 2017 13:21:19 GMT):
hold on while I lookit up
greg.haskins (Wed, 11 Oct 2017 13:21:26 GMT):
i cant remember the JIRA, searching now
joaquimpedrooliveira (Wed, 11 Oct 2017 13:21:31 GMT):
Thanks. I need to report this to my managers :)
joaquimpedrooliveira (Wed, 11 Oct 2017 13:22:03 GMT):
and I would like to track it on JIRA to know when it's resolved.
greg.haskins (Wed, 11 Oct 2017 13:22:45 GMT):
https://jira.hyperledger.org/browse/FAB-3996
greg.haskins (Wed, 11 Oct 2017 13:23:22 GMT):
if its the same issue, basically the system may not invalidate containers from a previous run
greg.haskins (Wed, 11 Oct 2017 13:23:42 GMT):
it usually materializes as a TLS problem since the old container has old certs
greg.haskins (Wed, 11 Oct 2017 13:24:03 GMT):
(But do note that the problem is deeper than just TLS, that is just how it surfaces)
greg.haskins (Wed, 11 Oct 2017 13:24:49 GMT):
I think @ecb is already working on a fix
ecb (Wed, 11 Oct 2017 13:24:49 GMT):
Has joined the channel.
joaquimpedrooliveira (Wed, 11 Oct 2017 13:30:50 GMT):
How can I request access to JIRA?
joaquimpedrooliveira (Wed, 11 Oct 2017 13:37:05 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=afaAYXSXvEs4gFpFs) @greg.haskins Is there anything I could look for in logs to be sure it's the same issue?
joaquimpedrooliveira (Wed, 11 Oct 2017 13:43:54 GMT):
In fact, I don't see in peers logs something similar to:
```2017-10-11 13:36:28.865 UTC [dockercontroller] Start -> DEBU 42a start-could not find image
joaquimpedrooliveira (Wed, 11 Oct 2017 13:44:31 GMT):
This was from trying to instantiate `marbles02`, that I've never run before
joaquimpedrooliveira (Wed, 11 Oct 2017 13:47:19 GMT):
But when I tried `fabcar`, that is the one I'm always using, this message is not shown
joaquimpedrooliveira (Wed, 11 Oct 2017 13:52:08 GMT):
Bingo! Using `marbles02` chaincode, that has never been used before, I could instantiate and invoke the chaincode!
joaquimpedrooliveira (Wed, 11 Oct 2017 13:52:29 GMT):
So I think I really bumped into `FAB-3996`.
greg.haskins (Wed, 11 Oct 2017 17:05:43 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=DAAnMWZmk85Kc8jWK) @rjones
rjones (Wed, 11 Oct 2017 17:05:43 GMT):
Has joined the channel.
rjones (Wed, 11 Oct 2017 18:29:38 GMT):
@joaquimpedrooliveira go to https://identity.linuxfoundation.org and create an account. Please do not use social logins - please register with an email account the first time. This will be your login to JIRA.
rjones (Wed, 11 Oct 2017 18:30:07 GMT):
Message Attachments
rjones (Wed, 11 Oct 2017 18:30:14 GMT):
@joaquimpedrooliveira ^^^
joaquimpedrooliveira (Wed, 11 Oct 2017 18:31:54 GMT):
@rjones, thanks!
rjones (Wed, 11 Oct 2017 18:32:47 GMT):
sure thing!
tsnyder (Wed, 11 Oct 2017 21:09:13 GMT):
Has joined the channel.
tsnyder (Thu, 12 Oct 2017 13:23:31 GMT):
@greg.haskins - I downloaded the make kubernetes you gave a link to on Oct 5th. When trying to build it I am receiving a build error - ../../vendor/github.com/miekg/pkcs11/pkcs11.go:29:18: fatal error: ltdl.h: No such file or directory. Per FAB-2684 this relates to the PKCS11 dynamic build that Vlad incorporated, but I am not sure how to resolve as I am testing the kubernetes and really do not want to add the overhead issues of dealing with the softhsm as I test. Suggestions on how to resolve?
mna2016 (Tue, 17 Oct 2017 12:40:16 GMT):
Has joined the channel.
mna2016 (Tue, 17 Oct 2017 12:42:48 GMT):
HI Team, My composer network was running fine till thursday last week. I was able to connect programatically to the composer network running on ibm container service(kubernetes). There was a new composer release last week. I also noticed that lot of script changes and yml changes were introduced on the github repo for container service. My question is: Do we need a new kind of connection profile to connect to the composer network? or would the old connection profile will do?
mna2016 (Tue, 17 Oct 2017 12:43:51 GMT):
Also, "kubectl proxy" does not work anymore. kubctl Ui can be useful for checking cluster logs etc.
mna2016 (Wed, 18 Oct 2017 12:02:20 GMT):
I was able to deploy the composer network on kubernetes container now. But kubectl proxy doesnt still work.
eclairamb (Wed, 18 Oct 2017 18:07:46 GMT):
Has joined the channel.
DannyWong (Fri, 20 Oct 2017 06:07:40 GMT):
Hey just wondering... I am running a standard 3 peers multi-host K8S with 100+ chaincode containers (they are created by peer with Docker Remote API, and not K8S pod)
I am wondering.... anyone has a better way to manage those cc containers? e.g. logs...etc
DannyWong (Fri, 20 Oct 2017 06:08:04 GMT):
can we change the logging driver of those chaincode containers?
DannyWong (Fri, 20 Oct 2017 10:29:39 GMT):
right now, i am thinking to change the default logging driver of the docker daemon such that all containers log will go to log management tool like Apache Spark
DannyWong (Fri, 20 Oct 2017 10:29:43 GMT):
Any idea from community?
greg.haskins (Fri, 20 Oct 2017 16:11:17 GMT):
@DannyWong generally speaking, there is interest in the community to make k8s type concerns more first class in the peer/chaincode model
greg.haskins (Fri, 20 Oct 2017 16:11:21 GMT):
that would include logging
greg.haskins (Fri, 20 Oct 2017 16:12:02 GMT):
so, if you have some thoughts on ways to make the logging more flexible, create a JIRA and start the conversation
DannyWong (Mon, 23 Oct 2017 06:02:37 GMT):
ok
luomin (Tue, 24 Oct 2017 07:37:21 GMT):
Has joined the channel.
RobertDiebels (Tue, 24 Oct 2017 13:02:04 GMT):
Has joined the channel.
greatyonchin (Wed, 25 Oct 2017 06:51:46 GMT):
Has joined the channel.
RobertDiebels (Wed, 25 Oct 2017 13:37:34 GMT):
Hey guys, any chance of this getting merged? https://gerrit.hyperledger.org/r/#/c/12159/
rjones (Wed, 25 Oct 2017 19:10:32 GMT):
@RobertDiebels not in current form, no. It has merge conflicts. @greg.haskins needs to de-conflict it and push a new version
rjones (Wed, 25 Oct 2017 19:10:59 GMT):
conflict.png
rjones (Wed, 25 Oct 2017 19:11:13 GMT):
@RobertDiebels the red "Cannot Merge" means there are conflicts
RobertDiebels (Wed, 25 Oct 2017 19:11:25 GMT):
I know. That's why I asked :P
rjones (Wed, 25 Oct 2017 19:12:00 GMT):
ah. well, that's on @greg.haskins (or one of the other k8s committers) to fix :)
RobertDiebels (Wed, 25 Oct 2017 19:12:37 GMT):
Ok thanks for the info :D
greg.haskins (Wed, 25 Oct 2017 19:13:04 GMT):
I’ll try to get it cleaned up asap
rjones (Wed, 25 Oct 2017 19:13:59 GMT):
it's weird, such a small change having conflicts
ericb 7 (Wed, 25 Oct 2017 22:49:55 GMT):
Has joined the channel.
GLB (Wed, 25 Oct 2017 23:48:35 GMT):
Has joined the channel.
GLB (Wed, 25 Oct 2017 23:55:12 GMT):
Hey everyone, I am trying to deploy my blockchain network to the free blockchain service on bluemix. I am following these instructions here: https://ibm-blockchain.github.io/interacting/ . I have deployed my business network in playground, and now I am having trouble accessing my rest API. I executed this command successfully as per the tutorial: 'create_composer-rest-server.sh --business-network-id
linzheng (Thu, 26 Oct 2017 04:16:08 GMT):
Has joined the channel.
RobertDiebels (Thu, 26 Oct 2017 08:14:26 GMT):
@greg.haskins @rjones thanks for the effort in advance guys :D !
baoyangc (Mon, 30 Oct 2017 03:35:43 GMT):
Has joined the channel.
d88 (Tue, 31 Oct 2017 19:56:35 GMT):
Has joined the channel.
jmcnevin (Wed, 01 Nov 2017 19:22:17 GMT):
anyone have examples of liveness probes on orderers/peers?
jmcnevin (Wed, 01 Nov 2017 19:23:36 GMT):
use tcpSocket probes tends to give me all sorts of broken pipe errors in my orderer debug logs, so I'm wondering if there's anything better to use
jeffgarratt (Wed, 01 Nov 2017 22:33:11 GMT):
@jmcnevin perhaps issue a config request to both?
jeffgarratt (Wed, 01 Nov 2017 22:33:45 GMT):
or a channel list for peer. Basically any CLI invoke will interface with any peer you wish
tsnyder (Thu, 02 Nov 2017 16:37:14 GMT):
Not sure where to ask this - but since I am using kubernetes minikube environment and setup is accordingly I thought I would start here and see if anyone had seen this and have a solution. Using the configuration, Dockerfile and kubernetes yamls I have been able to instantiate and orderer, 4 peers in 4 orgs, and a cli container. I have been able to successfully create a channel and have all peers join it. I have been able to install chaincode on all peers (example02). The last step initiating the chaincode is failing due a connection attempt timeout with the ordering service (see errs file). I have also attached my docker file for the peer0 and the kubernetes yaml. Any thoughts would be greatly appreciated.
tsnyder (Thu, 02 Nov 2017 16:39:55 GMT):
Sorry - I am getting an error saying media types are not accepted when trying to attach the files
tsnyder (Thu, 02 Nov 2017 16:43:13 GMT):
peer0yaml.txt
tsnyder (Thu, 02 Nov 2017 16:44:51 GMT):
errs.txt
tsnyder (Thu, 02 Nov 2017 16:49:13 GMT):
btw - the chaincode container gets successfully built
knagware9 (Thu, 02 Nov 2017 18:24:28 GMT):
@tsnyder --I think its due to fabric ca is not reachable...check ca node logs
darkcrux (Sat, 04 Nov 2017 03:10:31 GMT):
Has joined the channel.
qingsongGuo (Mon, 06 Nov 2017 06:33:38 GMT):
Has joined the channel.
Luke_Chen (Mon, 06 Nov 2017 13:57:04 GMT):
This blog introduces a way to deploy Fabric on Kubernetes. http://www.think-foundry.com/deploy-hyperledger-fabric-on-kubernetes-part-1/
Luke_Chen (Mon, 06 Nov 2017 13:57:19 GMT):
@here This blog introduces a way to deploy Fabric on Kubernetes. http://www.think-foundry.com/deploy-hyperledger-fabric-on-kubernetes-part-1/
DannyWong (Mon, 06 Nov 2017 14:00:37 GMT):
:thumbsup:
RobertDiebels (Mon, 06 Nov 2017 14:26:06 GMT):
@Luke_Chen Any chance you have the yaml resources for kubernetes available somewhere?
RobertDiebels (Mon, 06 Nov 2017 14:28:51 GMT):
Nvm found it. Didn't look through the fling long enough :D
RobertDiebels (Mon, 06 Nov 2017 14:29:22 GMT):
https://labs.vmware.com/flings/blockchain-on-vsphere#video <- found it there
RobertDiebels (Mon, 06 Nov 2017 14:39:42 GMT):
.tar file does contain the yaml files but IntelliJ can't find the right encoding for me.
RobertDiebels (Mon, 06 Nov 2017 14:40:20 GMT):
Tried loading it in UTF-8, 16 and windows-1252
Luke_Chen (Mon, 06 Nov 2017 14:40:23 GMT):
@RobertDiebels yes,these is a link to our fling, you can download the source code from the fling
Luke_Chen (Mon, 06 Nov 2017 14:40:44 GMT):
ok
Luke_Chen (Mon, 06 Nov 2017 14:41:26 GMT):
I wrote them at linux platform.
RobertDiebels (Mon, 06 Nov 2017 14:41:34 GMT):
Ah ok :P
Luke_Chen (Mon, 06 Nov 2017 14:47:52 GMT):
we will be thankful if you gave us some feedback :)
RobertDiebels (Mon, 06 Nov 2017 14:54:24 GMT):
I'm reading the pdf's now.
RobertDiebels (Mon, 06 Nov 2017 14:54:47 GMT):
But I would need the right encoding to get it to work on windows.
RobertDiebels (Mon, 06 Nov 2017 14:58:00 GMT):
@Luke_Chen nvm. I found my error. I opened the wrong files. I saw 12 files instead of 6. The wrong ones all had a prefix wtih "._" in it.
Luke_Chen (Mon, 06 Nov 2017 15:03:08 GMT):
@RobertDiebels all right, those are cache files
RobertDiebels (Mon, 06 Nov 2017 15:04:56 GMT):
This is exactly what I needed btw :D
Luke_Chen (Mon, 06 Nov 2017 15:05:42 GMT):
glad to help !
RobertDiebels (Mon, 06 Nov 2017 15:08:03 GMT):
Yea this is great! Will save me a bunch of work :) !
pmcosta1 (Mon, 06 Nov 2017 17:40:22 GMT):
Has joined the channel.
DannyWong (Tue, 07 Nov 2017 01:57:03 GMT):
Hi, I am setting 5 peers in one org (i.e. OrgA). They are configured as 5 deployment with 1 replica (not using StatefulSet). Each of them are having separate peer identity and crypto materials).
I have 100+ chaincode...
Problem Statement:
- If there are 2 peers scheduled to same K8S node (minion) then it will bring 200+ chaincode containers to that peers (and not managed by K8S). The memory and CPU of that host might be exhausted by this.
What I want...
- I want to ensure each peer will be scheduled by K8S to a K8S node (a.k.a minion) if there is no other peer on them
What I have done...
- Googled a bit on K8S scheduling (node/pod affinity and anti-affinity) and Stateful sets... Seems not get it working
Community, any help/insight?
DannyWong (Tue, 07 Nov 2017 01:57:03 GMT):
Hi, I am setting 5 peers in one org (i.e. OrgA). They are configured as 5 deployment with 1 replica (not using StatefulSet). Each of them are having separate peer identity and crypto materials).
I have 100+ chaincode...
Problem Statement:
- If there are 2 peers scheduled to same K8S node (minion) then it will bring 200+ chaincode containers to that node (and these workload not managed by K8S). The memory and CPU of that host might be exhausted by this.
What I want...
- I want to ensure each peer will be scheduled by K8S to a K8S node (a.k.a minion) if there is no other peer on them
What I have done...
- Googled a bit on K8S scheduling (node/pod affinity and anti-affinity) and Stateful sets... Seems not get it working
Community, any help/insight?
jmcnevin (Tue, 07 Nov 2017 13:39:17 GMT):
@DannyWong I've used anti-affinity rules to do this. What problems have you been seeing?
tsnyder (Tue, 07 Nov 2017 14:45:37 GMT):
@Luke_Chen Thanks for the blog ( http://www.think-foundry.com/deploy-hyperledger-fabric-on-kubernetes-part-1/ ). . It helped me get past the dns issue with a minikube for chaincode instantiation. Creating the minikube with the following cmd worked. minikube start --cpus 6 --memory 49152 --vm-driver kvm --docker-opt "dns=10.0.0.10 --dns=192.168.0.1 --dns-search \
default.svc.cluster.local --dns-search \
svc.cluster.local --dns-opt ndots:2 --dns-opt \
timeout:2 --dns-opt attempts:2 "
DannyWong (Wed, 08 Nov 2017 01:57:39 GMT):
@jmcnevin let me take a look... again
jmcnevin (Wed, 08 Nov 2017 22:31:52 GMT):
I'm wondering how everyone here is managing the docker VM that peers use. I've seen in many places that people are mounting the node's docker.sock and exposing that to the peer, but I'm wondering if this is allowing sibling containers to be created that could be orphaned if the peer pod is destroyed
jmcnevin (Wed, 08 Nov 2017 22:33:22 GMT):
i was kicking around the idea of running a completely new docker instance inside of the pod, next to the peer, and letting the peer use that instead, so all of its chaincode containers would be cleaned up along with the pod
jmcnevin (Wed, 08 Nov 2017 22:36:44 GMT):
this way you could also put resource requests around that docker host container depending on how much chaincode you expect to have installed per-peer
niteshsolanki (Thu, 09 Nov 2017 16:32:08 GMT):
Has joined the channel.
jskaqua (Thu, 09 Nov 2017 21:33:18 GMT):
Has joined the channel.
bjwswang (Sat, 11 Nov 2017 08:06:16 GMT):
Has joined the channel.
igors (Tue, 14 Nov 2017 20:34:29 GMT):
Has joined the channel.
daygee (Wed, 15 Nov 2017 13:09:33 GMT):
Hi guys
daygee (Wed, 15 Nov 2017 13:10:40 GMT):
so I have tried to deploy members of a hyperledger network to a kubernetes cluster
daygee (Wed, 15 Nov 2017 13:11:36 GMT):
I also implemented a kafka based ordering service
daygee (Wed, 15 Nov 2017 13:14:45 GMT):
but kafka and zookeeper seem to be acting up
daygee (Wed, 15 Nov 2017 13:16:40 GMT):
zookeeper keeps throwing: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
daygee (Wed, 15 Nov 2017 13:23:02 GMT):
and kafka is misbehaving so the orderer can't connect
daygee (Wed, 15 Nov 2017 13:23:19 GMT):
any suggestions would be appreciated
SwatiRaj (Thu, 16 Nov 2017 10:20:56 GMT):
Has joined the channel.
SwatiRaj (Fri, 17 Nov 2017 14:49:43 GMT):
Hi I was trying to deploy hyperledger on kubernetes , as you know kubernetes doesnot allow us to include '.'(dot) in service name , so my orderer service name is orderer-example-com . So I gave the orderer address as orderer-example-com:7050 , but when peer is trying to connect to orderer while initializing chaincode , I am getting this error - "Failed connecting to orderer-example-com:7050 , error: x509: certificate is valid for orderer.example.com, orderer, not orderer-example-com" , any help would be appreciated.
zlgonzalez (Sat, 18 Nov 2017 04:33:15 GMT):
Has joined the channel.
heidecke (Sun, 19 Nov 2017 09:25:20 GMT):
Has joined the channel.
SwatiRaj (Sun, 19 Nov 2017 20:56:04 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=c7dCCLJtruLa4kiFL) @tsnyder HEY tsnyder , since you have deployed orderer on Kubernetes , You must have changed the name of the orderer service from "." to something else , connect to that service not orderer.example.com
Luke_Chen (Mon, 20 Nov 2017 03:02:27 GMT):
@SwatiRaj maybe disable tls and try again
Riussi (Mon, 20 Nov 2017 08:51:02 GMT):
Has joined the channel.
daygee (Mon, 20 Nov 2017 10:58:47 GMT):
@SwatiRaj you have to change the orderer's name in the configtx file from orderer.example.com to orderer-example-com and then generate new certificates for it
daygee (Mon, 20 Nov 2017 11:00:55 GMT):
So apparently, I discovered that the zookeeper issue is something that can happen to zookeeper that doesn't necessarily stop functionality in the network
daygee (Mon, 20 Nov 2017 11:01:18 GMT):
Tho I still want to get to the bottom of it, I have left it for the time being
gentios (Mon, 20 Nov 2017 14:50:14 GMT):
Has joined the channel.
gentios (Mon, 20 Nov 2017 14:53:14 GMT):
Hi guys, I am new to deploying fabric + kafka with kubernetes
gentios (Mon, 20 Nov 2017 14:53:22 GMT):
can someone point me to some good resources
gentios (Mon, 20 Nov 2017 14:53:26 GMT):
on how to achieve that
RobertDiebels (Mon, 20 Nov 2017 14:55:20 GMT):
@gentios if you're on Linux this might be of interest: http://www.think-foundry.com/deploy-hyperledger-fabric-on-kubernetes-part-1
gentios (Mon, 20 Nov 2017 14:55:44 GMT):
@RobertDiebels thanks for that
gentios (Mon, 20 Nov 2017 14:56:05 GMT):
I have seen that, waiting for the second part
gentios (Mon, 20 Nov 2017 14:56:15 GMT):
since I don't have a lot of experience with kubernetes
RobertDiebels (Mon, 20 Nov 2017 14:56:46 GMT):
https://labs.vmware.com/flings/blockchain-on-vsphere#video <- might be helpful
gentios (Mon, 20 Nov 2017 14:57:05 GMT):
@RobertDiebels yes watching it now
gentios (Mon, 20 Nov 2017 14:57:15 GMT):
@RobertDiebels thanks for your honest support
RobertDiebels (Mon, 20 Nov 2017 14:57:49 GMT):
No problem.
gentios (Mon, 20 Nov 2017 14:58:44 GMT):
@RobertDiebels what if I don't want to go on with vSphere ?
RobertDiebels (Mon, 20 Nov 2017 14:58:50 GMT):
Maybe someone @here has some yaml-files ready to go. But the tarball you can download on the same page I linked, has some code which generates a Kubernetes yaml.
RobertDiebels (Mon, 20 Nov 2017 14:59:30 GMT):
If you download the tarball and the PDF from that page you'll have instructions on how to only generate the Kubernetes resources.
RobertDiebels (Mon, 20 Nov 2017 14:59:41 GMT):
I'm currently working on getting that working on Windows.
RobertDiebels (Mon, 20 Nov 2017 15:00:05 GMT):
(porting that code to NodeJS)
gentios (Mon, 20 Nov 2017 15:00:05 GMT):
@RobertDiebels thanks a lot
RobertDiebels (Mon, 20 Nov 2017 15:00:16 GMT):
No problem.
gentios (Tue, 21 Nov 2017 09:33:39 GMT):
@Luke_Chen Firstly, thank you for the great documentation and video tutorial
gentios (Tue, 21 Nov 2017 09:34:03 GMT):
I saw that in your video you have the kafka orderer files
gentios (Tue, 21 Nov 2017 09:34:35 GMT):
Can you share kafka based ordering service kubernete files if possible
Luke_Chen (Tue, 21 Nov 2017 10:03:11 GMT):
@gentios sorry about that, kafka cluster is not work as intended so far, thus we remove it from the fling, however, my colleague is doing this work recently, I will let you know immediately if he got any progress.
gentios (Tue, 21 Nov 2017 10:07:54 GMT):
ok thank you @Luke_Chen
gentios (Tue, 21 Nov 2017 10:08:02 GMT):
can you please share the fling link ?
Luke_Chen (Tue, 21 Nov 2017 10:10:33 GMT):
sure, https://labs.vmware.com/flings/blockchain-on-vsphere
gentios (Tue, 21 Nov 2017 10:11:54 GMT):
@Luke_Chen thank you
Luke_Chen (Tue, 21 Nov 2017 10:15:06 GMT):
@gentios welcome!
SwatiRaj (Tue, 21 Nov 2017 11:20:36 GMT):
@Luke_Chen it was working without tls , but I want it to work with tls enabled.
SwatiRaj (Tue, 21 Nov 2017 11:22:20 GMT):
@daygee Hey I already changed the name in configtx and it is working without tls , but with TLS , I need certificates for orderer-example-com , which cryptogen tool doesnt provide because the format in which cryptogen tool create certs is in
SwatiRaj (Tue, 21 Nov 2017 11:22:41 GMT):
SwatiRaj (Tue, 21 Nov 2017 11:23:09 GMT):
and I dont want any dot in the domain, because kubernetes doenot allow it .
gentios (Tue, 21 Nov 2017 14:09:31 GMT):
guys, with what tool have you generated the docker files into kubernetes
gentios (Tue, 21 Nov 2017 14:09:36 GMT):
while searching I saw this tool
gentios (Tue, 21 Nov 2017 14:09:37 GMT):
http://kompose.io/
daygee (Tue, 21 Nov 2017 14:56:34 GMT):
I used compose2kube https://github.com/kelseyhightower/compose2kube
daygee (Tue, 21 Nov 2017 15:05:20 GMT):
@SwatiRaj I see your problem
gentios (Tue, 21 Nov 2017 15:26:09 GMT):
thanks @daygee
daygee (Tue, 21 Nov 2017 15:27:11 GMT):
or you can always just do it yourself... :smirk:
daygee (Tue, 21 Nov 2017 15:27:38 GMT):
those tools may not get your config perfectly anyway
gentios (Tue, 21 Nov 2017 15:30:45 GMT):
yes I will see the both ways
gentios (Tue, 21 Nov 2017 15:31:00 GMT):
since I am new to kubernetes
Yixing (Wed, 22 Nov 2017 10:57:18 GMT):
Has joined the channel.
jackeyliliang (Fri, 24 Nov 2017 02:59:20 GMT):
Has joined the channel.
grapebaba (Fri, 24 Nov 2017 03:12:54 GMT):
hi guys
grapebaba (Fri, 24 Nov 2017 03:13:22 GMT):
[sarama] 2017/11/24 02:48:20.401886 client.go:397: client/brokers registered new broker #0 at kafka0-kafka-84db88d78d-2ggn7:9092
[sarama] 2017/11/24 02:48:20.401902 client.go:397: client/brokers registered new broker #1 at kafka1-kafka-74f4b6bbdf-8jbbv:9092
[sarama] 2017/11/24 02:48:20.401906 client.go:397: client/brokers registered new broker #2 at kafka2-kafka-586567dff8-xxjg2:9092
[sarama] 2017/11/24 02:48:20.401909 client.go:397: client/brokers registered new broker #3 at kafka3-kafka-559cd7bf5f-kpj2m:9092
[sarama] 2017/11/24 02:48:20.401931 client.go:154: Successfully initialized new client
2017-11-24
grapebaba (Fri, 24 Nov 2017 03:14:41 GMT):
anyone meet issues like sarama register broker use k8s pod name,it cause no host in dns
MohitYadav2317 (Fri, 24 Nov 2017 07:10:35 GMT):
Has joined the channel.
gentios (Fri, 24 Nov 2017 10:25:06 GMT):
@Luke_Chen, if for example I don't want to run in vSphere and just use an Amazon EC3 for example. I'll have to create a NFS there right ?
gentios (Fri, 24 Nov 2017 10:43:54 GMT):
I'll have to create a shared NFS/Amazon EFS, and than in a EC2/Kubernetes Pods
RobertDiebels (Sat, 25 Nov 2017 12:37:38 GMT):
@gentios this might be relevant for you. I'm currently attempting to drop the nfs server and use ConfigMaps bound to a persistentVolume to store the chain's configuration.
RobertDiebels (Sat, 25 Nov 2017 12:37:53 GMT):
That way you wouldn't have to create an NFS server.
LordGoodman (Sun, 26 Nov 2017 09:24:04 GMT):
Has joined the channel.
LordGoodman (Sun, 26 Nov 2017 09:45:19 GMT):
@gentios Of course you don't have to create a NFS, but you must make sure the every nodes can access their certificates and chain's configuration files.
sasiedu (Sun, 26 Nov 2017 09:54:31 GMT):
Has joined the channel.
gentios (Mon, 27 Nov 2017 07:19:46 GMT):
@Luke_Chen thank you for the second part
gentios (Mon, 27 Nov 2017 07:19:54 GMT):
you guys did a really good explanation
gentios (Mon, 27 Nov 2017 07:20:08 GMT):
if you need any help for documentation updates and stuff
gentios (Mon, 27 Nov 2017 07:20:12 GMT):
I am willing to contribute
gentios (Mon, 27 Nov 2017 07:54:12 GMT):
@LordGoodman why do you tink I don't have to create an NFS ?
daygee (Mon, 27 Nov 2017 09:45:30 GMT):
did anyone have issues with identities when deploying to kubernetes?
daygee (Mon, 27 Nov 2017 09:46:45 GMT):
I can't seem to understand why, but the orderer throws an error : Principal deserialization failure (The supplied identity is not valid, Verify() returned x509: certificate signed by unknown authority) for identity
Luke_Chen (Mon, 27 Nov 2017 15:52:49 GMT):
@gentios glad to help, and we are welcome for contribution !
Luke_Chen (Mon, 27 Nov 2017 15:53:43 GMT):
@gentios If we need help, will let you know at first. :)
Luke_Chen (Mon, 27 Nov 2017 15:57:35 GMT):
@daygee Have you tried regenerate all certificates?
Luke_Chen (Mon, 27 Nov 2017 16:08:24 GMT):
I think @LordGoodman means you could replace NFS with other shared storage solutions, since kubernetes support many of them. please refer this link https://kubernetes.io/docs/concepts/storage/volumes/
daygee (Mon, 27 Nov 2017 17:36:01 GMT):
that was my first guess but I checked all the certificates on the crypto-config folder and none of them match the one throwing exception
rjones (Mon, 27 Nov 2017 22:36:35 GMT):
Has left the channel.
Luke_Chen (Tue, 28 Nov 2017 02:49:17 GMT):
@daygee Where did you put all these certificates ? Did you use any shared storage solution?
daygee (Tue, 28 Nov 2017 06:34:51 GMT):
yes, after generating, I copied to a shared folder
daygee (Tue, 28 Nov 2017 07:04:52 GMT):
I checked an ssh decoder and the certificate is being generated for example.com meanwhile I have modified my configtx such that domain is no longer example.com for peer orgs and orderer
Luke_Chen (Tue, 28 Nov 2017 07:56:29 GMT):
could you send me your configtx.yaml and crypto-config.yaml ?
gentios (Tue, 28 Nov 2017 14:19:33 GMT):
@Luke_Chen I found a small inproovement in the tutorial: ```kubectl get pods –all-namespaces
gentios (Tue, 28 Nov 2017 14:19:33 GMT):
@Luke_Chen I found a small inproovement in the tutorial: ```kubectl get pods –all-namespaces should be ```kubectl get pods --all-namespaces
gentios (Tue, 28 Nov 2017 14:20:01 GMT):
should be ```kubectl get pods --all-namespaces
Luke_Chen (Tue, 28 Nov 2017 14:45:33 GMT):
@gentios Ah, apologize
gentios (Tue, 28 Nov 2017 14:47:33 GMT):
no no don't worry you've done the biggest part
gentios (Tue, 28 Nov 2017 14:48:05 GMT):
I just mention that to be the tutorial better
Luke_Chen (Tue, 28 Nov 2017 15:32:20 GMT):
@gentios will update it as soon as possible, thanks :)
gentios (Wed, 29 Nov 2017 08:38:26 GMT):
Guys, I tried to deploy the @Luke_Chen tutorial
gentios (Wed, 29 Nov 2017 08:38:36 GMT):
I created a kubernetes cluster with kubeadm
gentios (Wed, 29 Nov 2017 08:38:40 GMT):
installed flannel network
gentios (Wed, 29 Nov 2017 08:38:43 GMT):
created an nfs
gentios (Wed, 29 Nov 2017 08:38:56 GMT):
mounted the nfs
gentios (Wed, 29 Nov 2017 08:39:01 GMT):
and when I ran the script
gentios (Wed, 29 Nov 2017 08:39:08 GMT):
in the kubernetes cluster
gentios (Wed, 29 Nov 2017 08:39:18 GMT):
```org1 ca-7954ddf99b-hvmhc 0/1 ContainerCreating 0 4m
org1 cli-594b5c4d54-ctr8q 0/1 ContainerCreating 0 4m
org1 peer0-org1-58d9f4b5d9-mdbtd 0/2 ContainerCreating 0 4m
org1 peer1-org1-65cb9b676d-7kggx 0/2 ContainerCreating 0 4m
org2 ca-6797d754b-wzw8w 0/1 ContainerCreating 0 4m
org2 cli-6fdd6cc589-9ccbt 0/1 ContainerCreating 0 4m
org2 peer0-org2-89b7b4dfb-5bjxz 0/2 ContainerCreating 0 4m
org2 peer1-org2-55874dfccc-pc9px 0/2 ContainerCreating 0 4m
orgorderer1 orderer0-orgorderer1-5c6445c8db-5n65q 0/1 ContainerCreating 0 4m
gentios (Wed, 29 Nov 2017 08:39:29 GMT):
I see this
gentios (Wed, 29 Nov 2017 08:40:32 GMT):
the directory in the nfs is /home and is owned by nobody:nogroup
gentios (Wed, 29 Nov 2017 08:41:01 GMT):
Notice: I have ran this in the master node
gentios (Wed, 29 Nov 2017 08:41:14 GMT):
I don't have a worker node
Luke_Chen (Wed, 29 Nov 2017 09:27:01 GMT):
by default, we copy the channel-artifacts and crypto-config folder to /opt/share/
Luke_Chen (Wed, 29 Nov 2017 09:33:07 GMT):
By default, we use /opt/share folder of NFS to create Persistent Volumes
Luke_Chen (Wed, 29 Nov 2017 09:33:52 GMT):
meanwhile, we mount the NFS /opt/share folder to local /opt/share folder
gentios (Wed, 29 Nov 2017 09:34:28 GMT):
@Luke_Chen yes I noticed that now and I will fix it
Luke_Chen (Wed, 29 Nov 2017 09:34:33 GMT):
because generateAll.sh will copy the channel-artifacts and crypto-config folder to local /opt/share/
gentios (Wed, 29 Nov 2017 09:34:50 GMT):
can I run this in the master node ?
Luke_Chen (Wed, 29 Nov 2017 09:38:51 GMT):
Of course, since you have made master node schedulable.
gentios (Wed, 29 Nov 2017 09:46:44 GMT):
@Luke_Chen ok thanks a lot
gentios (Wed, 29 Nov 2017 09:46:52 GMT):
I will test it again and see
gentios (Wed, 29 Nov 2017 09:46:55 GMT):
what is happening
Luke_Chen (Wed, 29 Nov 2017 09:47:45 GMT):
you are welcome, feel free to contact.
Katiyman (Wed, 29 Nov 2017 09:48:09 GMT):
Has joined the channel.
Luke_Chen (Wed, 29 Nov 2017 16:02:38 GMT):
Hey guys, we have finished the blog about how to deploy Hyperledger Fabric on Kubernetes, if you want to deploy Fabric on pure Kubernetes environment, this will be a good place to start https://medium.com/@zhanghenry/how-to-deploy-hyperledger-fabric-on-kubernetes-2-751abf44c807 thanks.
sstone1 (Wed, 29 Nov 2017 19:51:36 GMT):
Has joined the channel.
aaron_z7 (Thu, 30 Nov 2017 05:40:47 GMT):
Has joined the channel.
tedljw (Thu, 30 Nov 2017 06:16:04 GMT):
Has joined the channel.
vsadriano (Thu, 30 Nov 2017 12:07:54 GMT):
Has joined the channel.
vsadriano (Thu, 30 Nov 2017 12:09:18 GMT):
Hi! Anybody run a Fabric Network on Kubernetes without root users on peers?
vsadriano (Thu, 30 Nov 2017 12:10:10 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=qRpdZJp7oQXownGse) @Luke_Chen There's no Orderer Type Kafka. :(
vsadriano (Thu, 30 Nov 2017 17:45:01 GMT):
Ok. It's great! How can I use "**unix:///host/var/run/docker.sock"**" on **CORE_VM_ENDPOINT** if my container user is non root? Any idea?
vsadriano (Thu, 30 Nov 2017 17:45:01 GMT):
Ok. It's great! How can I use "**unix:///host/var/run/docker.sock**" on **CORE_VM_ENDPOINT** if my container user is non root? Any idea?
vsadriano (Thu, 30 Nov 2017 17:45:01 GMT):
Ok. It's great! How can I use " **unix:///host/var/run/docker.sock** " on **CORE_VM_ENDPOINT** if my container user is non root? Any idea?
Luke_Chen (Fri, 01 Dec 2017 03:09:59 GMT):
@vsadriano Working on kafka type
Luke_Chen (Fri, 01 Dec 2017 03:11:04 GMT):
@vsadriano try to add the user to docker group
vsadriano (Fri, 01 Dec 2017 13:35:11 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=xDmEoY3t9SYA6fmpp) @Luke_Chen I don't like this approach (mapping docker.sock) but I'm searching for others. I'm seeing the VM settings. Thanks a lot!
Katiyman (Mon, 04 Dec 2017 08:03:26 GMT):
Hello Friends,
I am trying to setup hyperledger fabric on kuberneter using
but on running run.py.. my pods are not is the ready state and i get error
*Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/acdc1e23-d8c8-11e7-8b2d-d00d0d67dcb9/volumes/kubernetes.io~nfs/org1-pv --scope -- mount -t nfs 10.157.145.187:/opt/share/crypto-config/peerOrganizations/org1 /var/lib/kubelet/pods/acdc1e23-d8c8-11e7-8b2d-d00d0d67dcb9/volumes/kubernetes.io~nfs/org1-pv
Output: Running scope as unit run-19401.scope.
mount.nfs: access denied by server while mounting 10.157.145.187:/opt/share/crypto-config/peerOrganizations/org1*
i created the mount from one server and mounted it on all the server i can see the it properly mounted as well. i also ran
` chown nobody:nobody /opt/share-srvr`
unable to understand why this issue is coming
Kindly help
Katiyman (Mon, 04 Dec 2017 08:03:26 GMT):
Hello Friends,
I am trying to setup hyperledger fabric on kubernetes using
but on running run.py..
my pods are not is the ready state
i get error i all pods when i describe them
*Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/acdc1e23-d8c8-11e7-8b2d-d00d0d67dcb9/volumes/kubernetes.io~nfs/org1-pv --scope -- mount -t nfs 10.157.145.187:/opt/share/crypto-config/peerOrganizations/org1 /var/lib/kubelet/pods/acdc1e23-d8c8-11e7-8b2d-d00d0d67dcb9/volumes/kubernetes.io~nfs/org1-pv
Output: Running scope as unit run-19401.scope.
mount.nfs: access denied by server while mounting 10.157.145.187:/opt/share/crypto-config/peerOrganizations/org1*
i created the mount from one server and mounted it on all the server i can see the it properly mounted as well. i also ran
` chown nobody:nobody /opt/share-srvr`
unable to understand why this issue is coming
Kindly help
Katiyman (Mon, 04 Dec 2017 08:03:26 GMT):
Hello Friends,
I am trying to setup hyperledger fabric on kubernetes using
but on running run.py..
my pods are not is the ready state
i get error i all pods when i describe them
*Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/acdc1e23-d8c8-11e7-8b2d-d00d0d67dcb9/volumes/kubernetes.io~nfs/org1-pv --scope -- mount -t nfs 10.157.145.187:/opt/share/crypto-config/peerOrganizations/org1 /var/lib/kubelet/pods/acdc1e23-d8c8-11e7-8b2d-d00d0d67dcb9/volumes/kubernetes.io~nfs/org1-pv
Output: Running scope as unit run-19401.scope.
mount.nfs: access denied by server while mounting 10.157.145.187:/opt/share/crypto-config/peerOrganizations/org1*
i created the mount from one server and mounted it on all the server
i can see them it properly mounted as well.
i also ran
` chown nobody:nobody /opt/share-srvr`(centos)
unable to understand why this issue is coming
Kindly help
Katiyman (Mon, 04 Dec 2017 08:03:26 GMT):
Hello Friends,
I am trying to setup hyperledger fabric on kubernetes using
but on running run.py..
my pods are not is the ready state
i get error i all pods when i describe them
*Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/acdc1e23-d8c8-11e7-8b2d-d00d0d67dcb9/volumes/kubernetes.io~nfs/org1-pv --scope -- mount -t nfs 10.157.145.187:/opt/share/crypto-config/peerOrganizations/org1 /var/lib/kubelet/pods/acdc1e23-d8c8-11e7-8b2d-d00d0d67dcb9/volumes/kubernetes.io~nfs/org1-pv
Output: Running scope as unit run-19401.scope.
mount.nfs: access denied by server while mounting 10.157.145.187:/opt/share/crypto-config/peerOrganizations/org1*
i created the mount from one server and mounted it on all the server
i can see them it properly mounted as well.
i also ran
` chown nobody:nobody /opt/share-srvr`(centos)
unable to understand why this issue is coming
i can see PV like
kubectl get pv --all-namespaces
NAMESPACE NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
org1-artifacts-pv 500Mi RWX Retain Bound org1/org1-artifacts-pv 21m
org1-pv 500Mi RWX Retain Bound org1/org1-pv 21m
orgorderer1-pv 500Mi RWX Retain Bound orgorderer1/orgorderer1-pv 21m
i have only one org
Kindly help
Katiyman (Mon, 04 Dec 2017 08:03:26 GMT):
Hello Friends,
I am trying to setup hyperledger fabric on kubernetes using
http://www.think-foundry.com/deploy-hyperledger-fabric-on-kubernetes-part-2/
but on running run.py..
my pods are not is the ready state
i get error i all pods when i describe them
*Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/acdc1e23-d8c8-11e7-8b2d-d00d0d67dcb9/volumes/kubernetes.io~nfs/org1-pv --scope -- mount -t nfs 10.157.145.187:/opt/share/crypto-config/peerOrganizations/org1 /var/lib/kubelet/pods/acdc1e23-d8c8-11e7-8b2d-d00d0d67dcb9/volumes/kubernetes.io~nfs/org1-pv
Output: Running scope as unit run-19401.scope.
mount.nfs: access denied by server while mounting 10.157.145.187:/opt/share/crypto-config/peerOrganizations/org1*
i created the mount from one server and mounted it on all the server
i can see them it properly mounted as well.
i also ran
` chown nobody:nobody /opt/share-srvr`(centos)
unable to understand why this issue is coming
i can see PV like
kubectl get pv --all-namespaces
NAMESPACE NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
org1-artifacts-pv 500Mi RWX Retain Bound org1/org1-artifacts-pv 21m
org1-pv 500Mi RWX Retain Bound org1/org1-pv 21m
orgorderer1-pv 500Mi RWX Retain Bound orgorderer1/orgorderer1-pv 21m
i have only one org
Kindly help
atian15 (Mon, 04 Dec 2017 08:48:53 GMT):
Has joined the channel.
gentios (Mon, 04 Dec 2017 10:03:43 GMT):
@Katiyman have you changed the ip address of NFS in your YAML files ?
Katiyman (Mon, 04 Dec 2017 10:22:53 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=9dYSoWQvdRHSFf9cy) @gentios yes.. current i am reestablishing the NFS server even after tat if it doesnt work will get back
Katiyman (Mon, 04 Dec 2017 11:08:11 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=9dYSoWQvdRHSFf9cy) @gentios After creating the NFS volume do i need to mount it on all the worker node before running the run.py, (i have made the required changes in the yaml file)
Katiyman (Mon, 04 Dec 2017 11:47:08 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=9dYSoWQvdRHSFf9cy) @gentios After recreating the nfs volume i am not getting the mount issue. but getting below error
Normal Scheduled 6m default-scheduler Successfully assigned peer1-org1-65cb9b676d-m5ks5 to euca-10-254-113-59
Normal SuccessfulMountVolume 6m kubelet, euca-10-254-113-59 MountVolume.SetUp succeeded for volume "run"
Normal SuccessfulMountVolume 6m kubelet, euca-10-254-113-59 MountVolume.SetUp succeeded for volume "org1-pv"
Normal SuccessfulMountVolume 6m kubelet, euca-10-254-113-59 MountVolume.SetUp succeeded for volume "default-token-fd7zr"
Warning FailedCreatePodSandBox 6m (x5 over 6m) kubelet, euca-10-254-113-59 Failed create pod sandbox.
Warning DNSSearchForming 6m (x6 over 6m) kubelet, euca-10-254-113-59 Search Line limits were exceeded, some dns names have been omitted, the applied search line is: org1.svc.cluster.local svc.cluster.local cluster.local emea.nsn-net.net china.nsn-net.net apac.nsn-net.net
Warning FailedSync 6m (x6 over 6m) kubelet, euca-10-254-113-59 Error syncing pod
Normal SandboxChanged 1m (x107 over 6m) kubelet, euca-10-254-113-59 Pod sandbox changed, it will be killed and re-created.
Katiyman (Tue, 05 Dec 2017 04:56:05 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=PuQk8iNTTL9xahEBE) @gentios yes i have changed the ip addr in the cli and namespace yaml
Katiyman (Tue, 05 Dec 2017 09:57:01 GMT):
Hello All
below change
"--dns=10.0.0.10 --dns=192.168.0.1 --dns-search \
default.svc.cluster.local --dns-search \
svc.cluster.local --dns-opt ndots:2 --dns-opt \
timeout:2 --dns-opt attempts:2 "
is this supposed to be done in /etc/sysconfig/docker?
Katiyman (Tue, 05 Dec 2017 09:57:01 GMT):
Hello All
below change
"--dns=10.0.0.10 --dns=192.168.0.1 --dns-search \
default.svc.cluster.local --dns-search \
svc.cluster.local --dns-opt ndots:2 --dns-opt \
timeout:2 --dns-opt attempts:2 "
is this supposed to be done in /etc/sysconfig/docker?
Katiyman (Tue, 05 Dec 2017 09:57:01 GMT):
Hello All
in referece to
http://www.think-foundry.com/deploy-hyperledger-fabric-on-kubernetes-part-1/
below change
"--dns=10.0.0.10 --dns=192.168.0.1 --dns-search \
default.svc.cluster.local --dns-search \
svc.cluster.local --dns-opt ndots:2 --dns-opt \
timeout:2 --dns-opt attempts:2 "
is this supposed to be done in /etc/sysconfig/docker?
lvdh (Tue, 05 Dec 2017 13:42:09 GMT):
Has joined the channel.
gentios (Tue, 05 Dec 2017 14:41:25 GMT):
@Luke_Chen there are a lot of environment variables such as ```$namespace, $org etc...
gentios (Tue, 05 Dec 2017 14:41:31 GMT):
from where do they get the value
Luke_Chen (Tue, 05 Dec 2017 15:09:12 GMT):
please check the transform/config.py script, that script will render templates base on the architeture of crypto-config.
gentios (Tue, 05 Dec 2017 15:10:45 GMT):
thank you luke
Luke_Chen (Tue, 05 Dec 2017 15:13:25 GMT):
@Katiyman we added such docker options in /etc/default/docker, and I sure it will work in /etc/sysconfig/docker
samwood (Tue, 05 Dec 2017 20:12:27 GMT):
Has joined the channel.
Katiyman (Wed, 06 Dec 2017 04:35:33 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=yXtGSGS6yhKrGoLuw) @Luke_Chen Thanks Luke..
below is the content of my docker config file
OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false'
if [ -z "${DOCKER_CERT_PATH}" ]; then
DOCKER_CERT_PATH=/etc/docker
fi
# Do not add registries in this file anymore. Use /etc/containers/registries.conf
# from the atomic-registries package.
#
# docker-latest daemon can be used by starting the docker-latest unitfile.
# To use docker-latest client, uncomment below lines
#DOCKERBINARY=/usr/bin/docker-latest
#DOCKERDBINARY=/usr/bin/dockerd-latest
#DOCKER_CONTAINERD_BINARY=/usr/bin/docker-containerd-latest
#DOCKER_CONTAINERD_SHIM_BINARY=/usr/bin/docker-containerd-shim-latest
against which variable should i add tha config/
Luke_Chen (Wed, 06 Dec 2017 04:54:14 GMT):
Try to add such config to OPTIONS and restart the docker service
Luke_Chen (Wed, 06 Dec 2017 04:54:27 GMT):
@Katiyman
Katiyman (Wed, 06 Dec 2017 07:25:10 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=WxYWk5yTdGQcQXpCu) @Luke_Chen Thanks.. i put the value as
OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false --dns=10.244.0.2 --dns=192.168.0.1 --dns-search default.svc.cluster.local --dns-search svc.cluster.local --dns-opt ndots:2 --dns-opt timeout:2 --dns-opt attempts:2'
atian15 (Wed, 06 Dec 2017 09:38:00 GMT):
Has anyone encountered this problem?
atian15 (Wed, 06 Dec 2017 09:38:06 GMT):
2017-12-06 03:14:01.373 UTC [nodeCmd] createChaincodeServer -> WARN 03a peer.chaincodeListenAddress is not set, using peer0.org1-f-1:7052
2017-12-06 03:14:01.374 UTC [eventhub_producer] start -> INFO 03b Event processor started
panic: listen tcp 10.68.111.54:7052: bind: cannot assign requested address
goroutine 1 [running]:
github.com/hyperledger/fabric/peer/node.createChaincodeServer(0xc4200e2c00, 0x1dd, 0x3a0, 0xc4200142b2, 0xe, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hyperledger/fabric/peer/node/start.go:324 +0x52e
github.com/hyperledger/fabric/peer/node.serve(0x16417e8, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hyperledger/fabric/peer/node/start.go:166 +0x84c
github.com/hyperledger/fabric/peer/node.glob..func1(0x15c9e60, 0x16417e8, 0x0, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hyperledger/fabric/peer/node/start.go:89 +0x3f
github.com/hyperledger/fabric/vendor/github.com/spf13/cobra.(*Command).execute(0x15c9e60, 0x16417e8, 0x0, 0x0, 0x15c9e60, 0x16417e8)
/opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/spf13/cobra/command.go:599 +0x3e8
github.com/hyperledger/fabric/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x15ca4c0, 0xf, 0xc4200160f5, 0x7)
/opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/spf13/cobra/command.go:689 +0x2fe
github.com/hyperledger/fabric/vendor/github.com/spf13/cobra.(*Command).Execute(0x15ca4c0, 0x1b, 0xc4200160f5)
/opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/spf13/cobra/command.go:648 +0x2b
main.main()
/opt/gopath/src/github.com/hyperledger/fabric/peer/main.go:114 +0x50b
gentios (Wed, 06 Dec 2017 14:13:47 GMT):
@Luke_Chen it is used also couchdb as db, but I don't see any running pods or services ?
gentios (Wed, 06 Dec 2017 14:25:13 GMT):
also I am trying to install the chaincode via ```peer chaincode install -n mycc -v 1.0 –p github.com/hyperledger/fabric/peer/channel-artifacts/chaincode_example02
gentios (Wed, 06 Dec 2017 14:25:34 GMT):
but it is saying ``` Error getting chaincode code chaincode: path to chaincode does not exist:
gentios (Wed, 06 Dec 2017 14:26:04 GMT):
and the chaincode exists in /opt/share/channel-artifacts, and also in the cli in /channel-artifacts
Luke_Chen (Wed, 06 Dec 2017 14:26:43 GMT):
@gentios coubchdb container is running in peer pod, it doesn't need any services because it only need to connect peer whithin same pod.
gentios (Wed, 06 Dec 2017 14:27:39 GMT):
ok thanks for that @Luke_Chen
Luke_Chen (Wed, 06 Dec 2017 14:30:03 GMT):
Could show me output of `pwd` command in /channel-artifacts ？
Luke_Chen (Wed, 06 Dec 2017 14:30:16 GMT):
of cli container
Luke_Chen (Wed, 06 Dec 2017 14:30:33 GMT):
@gentios
gentios (Wed, 06 Dec 2017 14:31:13 GMT):
```root@cli-594b5c4d54-7crnd:/opt/gopath/src/github.com/hyperledger/fabric/peer# pwd
/opt/gopath/src/github.com/hyperledger/fabric/peer
root@cli-594b5c4d54-7crnd:/opt/gopath/src/github.com/hyperledger/fabric/peer#
gentios (Wed, 06 Dec 2017 14:31:31 GMT):
```root@cli-594b5c4d54-7crnd:/opt/gopath/src/github.com/hyperledger/fabric/peer/channel-artifacts# pwd
/opt/gopath/src/github.com/hyperledger/fabric/peer/channel-artifacts
gentios (Wed, 06 Dec 2017 14:31:55 GMT):
```root@cli-594b5c4d54-7crnd:/opt/gopath/src/github.com/hyperledger/fabric/peer/channel-artifacts# ls
Org1MSPanchors.tx Org2MSPanchors.tx chaincode_example02.go channel.tx genesis.block mychannel.block
gentios (Wed, 06 Dec 2017 14:32:40 GMT):
I am using this command to install the chaincode
gentios (Wed, 06 Dec 2017 14:32:43 GMT):
```peer chaincode install -n mycc -v 1.0 -p github.com/hyperledger/fabric/peer/channel-artifacts/chaincode_example02
Luke_Chen (Wed, 06 Dec 2017 14:33:14 GMT):
you need to create a chaincode_example02 dir under channel-artifacts, and `mv` chaincode_example02.go to chaincode_example02 dir
gentios (Wed, 06 Dec 2017 14:33:21 GMT):
ooo
gentios (Wed, 06 Dec 2017 14:33:33 GMT):
sorry I missed that
gentios (Wed, 06 Dec 2017 14:33:34 GMT):
:S
Luke_Chen (Wed, 06 Dec 2017 14:34:49 GMT):
@gentios haha not big deal
gentios (Wed, 06 Dec 2017 14:42:37 GMT):
@Luke_Chen thanks
Luke_Chen (Wed, 06 Dec 2017 14:43:26 GMT):
@gentios By the way, we are deploying fling v1.1 which will contain some new features like blockchain-explorer, kafka cluster and more flexiable way to define a fabric cluster, I will contact you once it was finished, maybe you could help us to refine some documents :)
gentios (Wed, 06 Dec 2017 14:45:56 GMT):
@Luke_Chen ofcourse I would be happy to contribute
gentios (Wed, 06 Dec 2017 14:47:00 GMT):
I can contribute and in code too
Luke_Chen (Wed, 06 Dec 2017 14:49:27 GMT):
@gentios Ok, great !
gentios (Thu, 07 Dec 2017 14:28:12 GMT):
in Ubuntu 16.04 where I would configure the ```"--dns=10.80.71.37 --dns=192.168.0.1 --dns-search default.svc.cluster.local --dns-search svc.cluster.local --dns-opt ndots:2 --dns-opt timeout:2 --dns-opt attempts:2"
gentios (Thu, 07 Dec 2017 15:06:46 GMT):
firstly I tried with a daemon.json in ```/etc/docker/daemon.json
gentios (Thu, 07 Dec 2017 15:07:00 GMT):
but that threw me some errors
gentios (Thu, 07 Dec 2017 15:07:16 GMT):
now I am trying with /etc/default/docker with DOCKER_OPTS
gentios (Thu, 07 Dec 2017 15:07:20 GMT):
hope this works
MohammadObaid (Thu, 07 Dec 2017 15:27:42 GMT):
Has joined the channel.
MohammadObaid (Thu, 07 Dec 2017 15:32:54 GMT):
I have seen multiple deplyoment options for fabric like deployment using ansible or with kubernetes. In terms of production and longer term which one is best and preferable
MohammadObaid (Thu, 07 Dec 2017 15:32:54 GMT):
I have seen multiple deplyoment options for fabric like deployment using ansible or with kubernetes or with swarm. In terms of production and longer term which one is best and preferable
gentios (Thu, 07 Dec 2017 15:40:38 GMT):
@MohammadObaid kubernetes
MohammadObaid (Thu, 07 Dec 2017 15:56:03 GMT):
@gentios I am following this tutorial to setup using kubernetes . http://www.think-foundry.com/deploy-hyperledger-fabric-on-kubernetes-part-2/ . This article assumes that we already setup kubernetes between multiple hosts right ? Any other tutorial you would recommend me?
Luke_Chen (Fri, 08 Dec 2017 01:37:51 GMT):
@MohammadObaid If you need some articles to setup kubernetes, you better take a look of their website https://kubernetes.io/
Luke_Chen (Fri, 08 Dec 2017 01:41:37 GMT):
@gentios After restarted the docker service, you can use command `ps -aux | grep docker` to check weather your config has been commited in docker daemon
Luke_Chen (Fri, 08 Dec 2017 01:41:37 GMT):
@gentios After restart the docker service, you can use command `ps -aux | grep docker` to check wether your config has been commited in docker daemon
Luke_Chen (Fri, 08 Dec 2017 01:41:37 GMT):
@gentios After restart the docker service, you can use command `ps -aux | grep docker` to check weather your config has been commited in docker daemon
Katiyman (Fri, 08 Dec 2017 04:39:41 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=EfgvAzhHAJYqpzCAs) @gentios [ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=EfgvAzhHAJYqpzCAs) @gentios i put it in /etc/sysconfig/docker for centos with OPTIONS variable
Katiyman (Fri, 08 Dec 2017 04:39:41 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=EfgvAzhHAJYqpzCAs) @gentios [ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=EfgvAzhHAJYqpzCAs) @gentios i put it in /etc/sysconfig/docker for OPTIONS variable
JayJong (Fri, 08 Dec 2017 11:13:30 GMT):
Has joined the channel.
gentios (Fri, 08 Dec 2017 13:47:01 GMT):
@Katiyman what about in ubuntu ?
Luke_Chen (Fri, 08 Dec 2017 15:33:11 GMT):
@gentios Have you tried /etc/default/docker ?
alvaradojl (Sun, 10 Dec 2017 12:23:00 GMT):
Has joined the channel.
Katiyman (Mon, 11 Dec 2017 03:57:45 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=TxJLJRGQcrXwJy49q) @gentios I have not tried on the ubuntu but as pointed by Luke it should be /etc/default/docker to DOCKER_OPTS var
gentios (Mon, 11 Dec 2017 09:54:52 GMT):
exit
gentios (Mon, 11 Dec 2017 09:57:11 GMT):
@Luke_Chen yes I have don that and I am getting timout expired
gentios (Mon, 11 Dec 2017 09:57:11 GMT):
yes I have don that and I am getting timout expired
gentios (Mon, 11 Dec 2017 09:57:13 GMT):
```Error: Error endorsing chaincode: rpc error: code = Unknown desc = Timeout expired while starting chaincode mycc:1.0(networkid:dev,peerid:peer0.org1,tx:8d5ec6bab50ba69a202944f3b802aa86b3d4db0a0dffcfaf6327b3444af6d917)
gentios (Mon, 11 Dec 2017 09:58:10 GMT):
my ```DOCKER_OPTS="--dns=10.80.70.214 --dns=192.168.0.1 --dns-search default.svc.cluster.local --dns-search svc.cluster.local --dns-opt ndots:2 --dns-opt timeout:2 --dns-opt attempts:2 "
gentios (Mon, 11 Dec 2017 09:58:24 GMT):
and the kubedns
gentios (Mon, 11 Dec 2017 09:58:27 GMT):
```KubeDNS is running at https://10.80.70.214:6443/api/v1/namespaces/kube-system/services/kube-dns/proxy
gentios (Mon, 11 Dec 2017 09:59:40 GMT):
kc
gentios (Mon, 11 Dec 2017 10:08:26 GMT):
I am testing if the flannel network is configured correctly
gentios (Mon, 11 Dec 2017 10:35:47 GMT):
nope still no luck to deploy a chaincode
gentios (Mon, 11 Dec 2017 10:35:58 GMT):
I will keep testing in why this happens
gentios (Mon, 11 Dec 2017 10:53:01 GMT):
the log in the container is ```2017-12-11 10:51:39.269 UTC [chaincode] Launch -> ERRO 411 launchAndWaitForRegister failed Timeout expired while starting chaincode mycc:1.0(networkid:dev,peerid:peer0.org1,tx:5ded7b093b18ad228fd80c036bdfc40917baabe88a2ce8e619ec25c89211c438)
Luke_Chen (Mon, 11 Dec 2017 11:09:09 GMT):
@gentios chaincode container exit once it was created, did you meet this situation?
Luke_Chen (Mon, 11 Dec 2017 11:10:17 GMT):
what's the output when you run `peer node status` in cli container
Katiyman (Mon, 11 Dec 2017 12:27:29 GMT):
Hello one ques... the chain code docker container starts when i instantiate the chain code. and that is not managed by kubernetes. is there a way to bring it in kubernetes
Luke_Chen (Mon, 11 Dec 2017 13:28:24 GMT):
@Katiyman Good question ! No way so far
Katiyman (Tue, 12 Dec 2017 04:28:55 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=zPyNmcjytGWMGCjTR) @gentios how are you finding the kube-dns ip addr?
guolidong (Tue, 12 Dec 2017 06:03:06 GMT):
Has joined the channel.
gentios (Tue, 12 Dec 2017 07:27:20 GMT):
@Katiyman ```kubectl cluster-info
gentios (Tue, 12 Dec 2017 07:27:37 GMT):
did you managed to deploy a chaincode based on Luke tutorial ?
Katiyman (Tue, 12 Dec 2017 08:03:17 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=Mqu3pWJH33BzSHHr8) @gentios Yes i did... i am not sure if there are other ways but the way i found the kube-dns pod is byt describing the kube-dns pod then using ip adddr that is coming.
kubectl describe po kube-dns-545bc4bfd4-d2bzv -n kube-system
Name: kube-dns-545bc4bfd4-d2bzv
Namespace: kube-system
Node: euca-10-254-107-23.eucalyptus.internal/10.157.145.187
Start Time: Wed, 06 Dec 2017 09:12:32 +0200
Labels: k8s-app=kube-dns
pod-template-hash=1016706980
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kube-system","name":"kube-dns-545bc4bfd4","uid":"f7a66398-da53-11e7-bc28-d00d0d67...
Status: Running
IP: 10.244.0.2
Created By: ReplicaSet/kube-dns-545bc4bfd4
Controlled By: ReplicaSet/kube-dns-545bc4bfd4
gentios (Tue, 12 Dec 2017 08:08:43 GMT):
@Katiyman maybe you can achieve the same results and with this have to try it
gentios (Tue, 12 Dec 2017 08:08:59 GMT):
anyway how did you create the kubernetes cluster
gentios (Tue, 12 Dec 2017 08:09:02 GMT):
with what tool ?
gentios (Tue, 12 Dec 2017 08:09:42 GMT):
because I did use the kubeadm and couldn't deploy a chaincode
Katiyman (Tue, 12 Dec 2017 08:26:01 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=E5HidpSPpXCnrNeDH) @gentios i also used kubeadm
gentios (Tue, 12 Dec 2017 08:26:20 GMT):
hmm
ysim (Wed, 13 Dec 2017 20:25:56 GMT):
Submitted a CR that will traverse symlinks for pem material, making it possible to use Kubernetes Secrets for the crypto material: https://gerrit.hyperledger.org/r/#/c/16133 Would be awesome to get some eyes on it.
lvdh (Thu, 14 Dec 2017 10:37:45 GMT):
@ysim Good stuff, thanks! I know one guy who would love to see that accepted/merged https://medium.com/wearetheledger/a-first-attempt-at-hyperledger-fabric-kubernetes-66e43b12a211
RobertDiebels (Thu, 14 Dec 2017 10:49:00 GMT):
@ysim I would be extremely happy when that gets in. @lvdh I ran into the same problem yesterday. Didn't know that blog existed unfortunately.
vvnick (Thu, 14 Dec 2017 11:09:33 GMT):
Has joined the channel.
nitin6ul (Sat, 16 Dec 2017 17:55:33 GMT):
Has joined the channel.
zhishui (Tue, 19 Dec 2017 08:11:00 GMT):
Has joined the channel.
sravs (Wed, 20 Dec 2017 06:16:04 GMT):
Has joined the channel.
luminance (Wed, 20 Dec 2017 08:56:20 GMT):
Has joined the channel.
Katiyman (Wed, 20 Dec 2017 09:32:43 GMT):
Hello
I see below env variables exposed in all the containers .. how are there getting referenced...?
Katiyman (Wed, 20 Dec 2017 09:32:43 GMT):
Hello
I see below env variables exposed in all the containers .. how are there getting referenced...?
PEER6_PORT_7051_TCP=tcp://10.111.56.13:7051
PEER2_PORT=tcp://10.99.166.99:7051
PEER6_SERVICE_PORT_CHAINCODE_LISTEN=7052
PEER3_SERVICE_PORT=7051
PEER2_SERVICE_HOST=10.99.166.99
HOSTNAME=peer0-org1-58d9f4b5d9-sp7c6
CA_PORT_7054_TCP_PROTO=tcp
PEER5_PORT_7051_TCP_PROTO=tcp
PEER3_PORT_7051_TCP_PORT=7051
PEER1_SERVICE_PORT=7051
PEER0_SERVICE_HOST=10.96.133.98
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT=tcp://10.96.0.1:443
PEER2_SERVICE_PORT=7051
CORE_PEER_TLS_ROOTCERT_FILE=/etc/hyperledger/fabric/tls/ca.crt
PEER6_SERVICE_PORT_EXTERNALE_LISTEN_ENDPOINT=7051
PEER6_PORT_7051_TCP_PORT=7051
PEER1_SERVICE_PORT_CHAINCODE_LISTEN=7052
KUBERNETES_SERVICE_PORT=443
PEER1_PORT_7051_TCP_ADDR=10.100.108.19
CORE_PEER_TLS_KEY_FILE=/etc/hyperledger/fabric/tls/server.key
PEER5_SERVICE_PORT_EXTERNALE_LISTEN_ENDPOINT=7051
KUBERNETES_SERVICE_HOST=10.96.0.1
PEER0_PORT_7052_TCP=tcp://10.96.133.98:7052
PEER3_PORT_7051_TCP_PROTO=tcp
PEER0_PORT=tcp://10.96.133.98:7051
PEER2_PORT_7051_TCP_PORT=7051
PEER0_PORT_7052_TCP_PORT=7052
LS_COLORS=
PEER5_PORT_7052_TCP_ADDR=10.98.185.31
CA_PORT=tcp://10.101.141.34:7054
PEER6_PORT_7051_TCP_PROTO=tcp
PEER4_PORT_7051_TCP_PORT=7051
PEER3_SERVICE_PORT_CHAINCODE_LISTEN=7052
PEER2_PORT_7051_TCP=tcp://10.99.166.99:7051
CORE_PEER_PROFILE_ENABLED=true
CORE_PEER_GOSSIP_ORGLEADER=false
PEER4_PORT_7052_TCP_ADDR=10.96.130.180
PEER4_PORT_7051_TCP_PROTO=tcp
CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=localhost:5984
PEER2_PORT_7052_TCP_ADDR=10.99.166.99
PEER0_SERVICE_PORT_EXTERNALE_LISTEN_ENDPOINT=7051
PEER4_SERVICE_PORT_CHAINCODE_LISTEN=7052
PEER2_PORT_7051_TCP_PROTO=tcp
PEER6_PORT_7052_TCP_PORT=7052
CORE_PEER_LOCALMSPID=Org1MSP
PEER1_PORT_7052_TCP=tcp://10.100.108.19:7052
PEER1_SERVICE_PORT_EXTERNALE_LISTEN_ENDPOINT=7051
PEER0_PORT_7051_TCP_PROTO=tcp
PEER0_PORT_7052_TCP_ADDR=10.96.133.98
CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
PEER4_SERVICE_PORT_EXTERNALE_LISTEN_ENDPOINT=7051
CA_PORT_7054_TCP_PORT=7054
PEER6_PORT_7052_TCP=tcp://10.111.56.13:7052
PEER5_PORT_7051_TCP_PORT=7051
PEER1_PORT_7052_TCP_PROTO=tcp
CORE_PEER_TLS_CERT_FILE=/etc/hyperledger/fabric/tls/server.crt
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
CA_SERVICE_PORT_ENDPOINT=7054
PEER5_SERVICE_PORT_CHAINCODE_LISTEN=7052
PEER5_SERVICE_HOST=10.98.185.31
PEER5_PORT_7051_TCP=tcp://10.98.185.31:7051
PEER1_PORT_7051_TCP_PORT=7051
CA_PORT_7054_TCP=tcp://10.101.141.34:7054
PWD=/opt/gopath/src/github.com/hyperledger/fabric/peer
PEER5_SERVICE_PORT=7051
PEER4_SERVICE_HOST=10.96.130.180
PEER3_PORT_7052_TCP=tcp://10.105.216.216:7052
PEER1_SERVICE_HOST=10.100.108.19
PEER0_SERVICE_PORT_CHAINCODE_LISTEN=7052
PEER3_PORT=tcp://10.105.216.216:7051
PEER2_PORT_7052_TCP_PROTO=tcp
PEER5_PORT=tcp://10.98.185.31:7051
PEER2_SERVICE_PORT_EXTERNALE_LISTEN_ENDPOINT=7051
CA_SERVICE_PORT=7054
PEER0_PORT_7051_TCP_PORT=7051
CORE_PEER_TLS_ENABLED=false
PEER4_PORT_7052_TCP=tcp://10.96.130.180:7052
PEER1_PORT=tcp://10.100.108.19:7051
CA_SERVICE_HOST=10.101.141.34
PEER3_PORT_7052_TCP_PORT=7052
CORE_PEER_ID=peer0.org1
SHLVL=1
HOME=/root
PEER3_PORT_7052_TCP_PROTO=tcp
PEER4_PORT_7052_TCP_PORT=7052
PEER0_PORT_7051_TCP=tcp://10.96.133.98:7051
PEER4_SERVICE_PORT=7051
PEER5_PORT_7052_TCP_PORT=7052
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_PORT_HTTPS=443
PEER6_PORT_7052_TCP_PROTO=tcp
CORE_LOGGING_LEVEL=DEBUG
PEER3_PORT_7051_TCP=tcp://10.105.216.216:7051
CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer0.org1:7051
FABRIC_CFG_PATH=/etc/hyperledger/fabric
PEER0_SERVICE_PORT=7051
PEER2_PORT_7051_TCP_ADDR=10.99.166.99
PEER1_PORT_7052_TCP_PORT=7052
PEER1_PORT_7051_TCP=tcp://10.100.108.19:7051
PEER6_PORT_7052_TCP_ADDR=10.111.56.13
PEER4_PORT_7051_TCP_ADDR=10.96.130.180
PEER6_SERVICE_PORT=7051
PEER5_PORT_7052_TCP=tcp://10.98.185.31:7052
CORE_PEER_ADDRESS=peer0.org1:7051
PEER4_PORT_7052_TCP_PROTO=tcp
PEER1_PORT_7052_TCP_ADDR=10.100.108.19
CA_PORT_7054_TCP_ADDR=10.101.141.34
PEER5_PORT_7051_TCP_ADDR=10.98.185.31
PEER5_PORT_7052_TCP_PROTO=tcp
PEER3_SERVICE_HOST=10.105.216.216
PEER0_PORT_7051_TCP_ADDR=10.96.133.98
PEER1_PORT_7051_TCP_PROTO=tcp
CORE_LEDGER_STATE_STATEDATABASE=CouchDB
PEER4_PORT=tcp://10.96.130.180:7051
PEER4_PORT_7051_TCP=tcp://10.96.130.180:7051
PEER0_PORT_7052_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
CORE_PEER_GOSSIP_USELEADERELECTION=true
PEER2_PORT_7052_TCP_PORT=7052
PEER6_PORT=tcp://10.111.56.13:7051
PEER3_PORT_7052_TCP_ADDR=10.105.216.216
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
PEER3_PORT_7051_TCP_ADDR=10.105.216.216
PEER2_SERVICE_PORT_CHAINCODE_LISTEN=7052
PEER3_SERVICE_PORT_EXTERNALE_LISTEN_ENDPOINT=7051
PEER6_PORT_7051_TCP_ADDR=10.111.56.13
PEER6_SERVICE_HOST=10.111.56.13
PEER2_PORT_7052_TCP=tcp://10.99.166.99:7052
_=/usr/bin/printenv
luminance (Wed, 20 Dec 2017 09:52:12 GMT):
Hey guys, has anyone deployed fabric on on-premises kubernetes?
I have created the whole network (2 orgs + 1 orderer org), but when I create a chaincode (with composer), the container (with the chaincode) cannot connect to the kubernetes peer nodes. I have also setup dokcer host (that is the kubernetes worker node) with --dns and --dns-search that will match the kubernetes DNS service. Any idea how to bypass this issue? What should I set in the docker engine? Is there something else that I am missing?
luminance (Wed, 20 Dec 2017 09:52:12 GMT):
Hey guys, has anyone deployed fabric on on-premises kubernetes?
I have created the whole network (2 orgs + 1 orderer org), but when I create a chaincode (with composer), the container (with the chaincode) cannot connect to the kubernetes peer nodes. I have also setup dokcer host (that is the kubernetes worker node) with --dns and --dns-search that will match the kubernetes DNS service. Now when I start the network (with composer), the chaincode (dev-peer0.org1..........) containers are created. but it hits me with this error.
```
⠙ Starting business network definition. This may take a minute...(node:29194) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: The event hub has not been connected to the event source
(node:29194) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:29194) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 2): Error: The event hub has not been connected to the event source
✖ Starting business network definition. This may take a minute...
Error: Error trying to instantiate composer runtime. Error: The event hub has not been connected to the event source
Command failed
```
Any idea how to bypass this issue? What should I set in the docker engine? Is there something else that I am missing?
luminance (Wed, 20 Dec 2017 09:58:16 GMT):
@Katiyman The parameters are not relevant if you are not running any command from the containers itself. They are just ENV Variable holders, so when you run `peer create` or `peer join` you will already have the needed ENV VARs in the background and the commands are executed based on that.
Varun2887 (Wed, 20 Dec 2017 10:09:23 GMT):
Has joined the channel.
Varun2887 (Wed, 20 Dec 2017 10:09:43 GMT):
is fabric setup using kubernetes production ready?
prabvi01 (Wed, 20 Dec 2017 10:41:36 GMT):
Has joined the channel.
luminance (Wed, 20 Dec 2017 11:19:44 GMT):
@Varun2887 Do you know any setup (repo) that is not production ready? Have you tried any?
Varun2887 (Wed, 20 Dec 2017 11:20:14 GMT):
i started on cellos but it says its not production ready
luminance (Wed, 20 Dec 2017 11:22:14 GMT):
@Varun2887 I think it is far from ready. I have tried it also, but with no luck. Did you manage to instantiate a chaincode?
Varun2887 (Wed, 20 Dec 2017 11:22:54 GMT):
I stopped using it when i saw its not full fledge prod supported
Varun2887 (Wed, 20 Dec 2017 11:23:26 GMT):
what shall be the best way to use it, assuming we have to deploy it on prod as well
Varun2887 (Wed, 20 Dec 2017 11:23:35 GMT):
kubernetes setup works fine?
Varun2887 (Wed, 20 Dec 2017 11:23:40 GMT):
on multiple machine?
MR (Wed, 20 Dec 2017 12:26:09 GMT):
Has joined the channel.
daygee (Wed, 20 Dec 2017 13:38:56 GMT):
Hi everyone
daygee (Wed, 20 Dec 2017 13:39:40 GMT):
so I have a 1 peer- 1 org network up on kubernetes
daygee (Wed, 20 Dec 2017 13:39:49 GMT):
can create channel
daygee (Wed, 20 Dec 2017 13:39:53 GMT):
join channel
daygee (Wed, 20 Dec 2017 13:39:58 GMT):
install chaincode
daygee (Wed, 20 Dec 2017 13:40:07 GMT):
but I can't instantiate chaincode
daygee (Wed, 20 Dec 2017 13:40:57 GMT):
I believe the issue is in the way the peer tries to pull fabric-ccenv image from public docker repo
daygee (Wed, 20 Dec 2017 13:41:40 GMT):
Clipboard - December 20, 2017 2:41 PM
daygee (Wed, 20 Dec 2017 13:43:05 GMT):
I was thinking I can maybe find out how to configure the peer to pull the image from my company repo as trying to pull from an external repo always fails
luminance (Wed, 20 Dec 2017 13:48:40 GMT):
@daygee That means that the dev-* containers are up and running and connected with to the peer?
daygee (Wed, 20 Dec 2017 14:02:54 GMT):
the dev container never gets created
luminance (Wed, 20 Dec 2017 14:09:50 GMT):
@daygee You need to setup the docker host Daemon config to refer to the Kubernetes DNS service IP address:
```
kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.96.0.1
luminance (Wed, 20 Dec 2017 14:09:50 GMT):
@daygee You need to setup the docker host Daemon config to refer to the Kubernetes DNS service IP address:
```
kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.96.0.1
luminance (Wed, 20 Dec 2017 14:11:24 GMT):
@daygee BTW, I am stuck on the same issue. The container gets created, but it never connects to the kubernetes virtual network of the namespaces.
daygee (Wed, 20 Dec 2017 14:15:12 GMT):
so docker points to kubernetes to deploy the containers?
luminance (Wed, 20 Dec 2017 14:20:22 GMT):
@daygee Well, because the instantiating of a chaincode actually creates a new container on the Docker host, but that container is not connected to the Kubernetes network(namespace) in which your org/peer exists. So the only way is for the newly created container to communicate with the peer is to set-up the DNS.
daygee (Wed, 20 Dec 2017 14:21:10 GMT):
okay, I understand
daygee (Wed, 20 Dec 2017 14:21:51 GMT):
so when you set up the dns, [ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=LaNZ9ht9YWAJEb3Me) @luminance this became your issue?
luminance (Wed, 20 Dec 2017 14:27:32 GMT):
@daygee Probably. I am not sure what is the issue actually. That is why I asked few question here and on #fabric channel, but with no avail.
Katiyman (Thu, 21 Dec 2017 07:36:34 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=5xoqN4TtFAxYeycrd) @luminance [ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=5xoqN4TtFAxYeycrd) @luminance Thanks for the reply.. but aren't these command to be executed from the cli container? if yes then i think these vars are not req in the peer containers
luminance (Thu, 21 Dec 2017 11:45:38 GMT):
Question regarding k8s setup. Is [this](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=t7DjfyzC9Eopu6cYo) still valid (with fabric1.0.5) ?
When setting those variables, i am hit with an error when trying to `composer runtime install`:
```
✖ Installing runtime for business network musala-blockchain. This may take a minute...
Error: Error trying install composer runtime. Error: No valid responses from any peers.
Response from attempted peer comms was an error: Error: Failed to deserialize creator identity, err The supplied identity is not valid, Verify() returned x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "ca.org1")
Response from attempted peer comms was an error: Error: Failed to deserialize creator identity, err The supplied identity is not valid, Verify() returned x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "ca.org1")
```
misaelssantos (Thu, 21 Dec 2017 17:53:42 GMT):
Has joined the channel.
xeonpitar (Mon, 25 Dec 2017 02:12:09 GMT):
Has joined the channel.
olufemiisrael (Thu, 28 Dec 2017 07:30:14 GMT):
Has joined the channel.
shanlusun (Fri, 29 Dec 2017 02:19:53 GMT):
Has joined the channel.
shenhiboy (Fri, 29 Dec 2017 07:08:46 GMT):
Has joined the channel.
Kaltrak (Fri, 29 Dec 2017 12:52:31 GMT):
Has joined the channel.
marc0o (Sat, 30 Dec 2017 19:27:34 GMT):
Has joined the channel.
elias_p (Tue, 02 Jan 2018 22:57:56 GMT):
Has joined the channel.
gurel (Wed, 03 Jan 2018 17:43:37 GMT):
Has joined the channel.
daygee (Thu, 04 Jan 2018 10:40:06 GMT):
Hello
daygee (Thu, 04 Jan 2018 10:40:29 GMT):
I've been thinking about the instantiation of chaincode
daygee (Thu, 04 Jan 2018 10:40:35 GMT):
and how problematic it is
daygee (Thu, 04 Jan 2018 10:42:46 GMT):
I was wondering, isn't it possible for the peer pod to create a ccenv pod in kubernetes, rather than create a docker container on the node
daygee (Thu, 04 Jan 2018 10:43:08 GMT):
or at least make it an option
Luke_Chen (Thu, 04 Jan 2018 10:48:09 GMT):
@daygee no such option so far, the only way to created a cc container is by docker daemon of the node rather than kubernetes api.
daygee (Thu, 04 Jan 2018 10:49:02 GMT):
is there any potential that it will be a task to do?
daygee (Thu, 04 Jan 2018 10:50:26 GMT):
I ask because it seems like too much work from a dev ops point of view to have to manage the kube-dns for each node
daygee (Thu, 04 Jan 2018 10:51:19 GMT):
also, in the event a peer dies and is recreated in another node, because kubernetes doesn't know about the container, someone has to go shut it down in that node
daygee (Thu, 04 Jan 2018 10:51:50 GMT):
and if the node the new peer has been created on will have to be configured
daygee (Thu, 04 Jan 2018 10:51:50 GMT):
and then the node the new peer has been created on will have to be configured
daygee (Thu, 04 Jan 2018 10:54:02 GMT):
whereas, kubernetes manages all this for you
Luke_Chen (Thu, 04 Jan 2018 11:01:32 GMT):
Yes, I can't agree with you point any more, which is a huge disadvantage for deploying Fabric on kubernetes so far.
Luke_Chen (Thu, 04 Jan 2018 11:09:45 GMT):
meanwhile, I am sure it should be a good idea to add feature like supporting kubernetes in Fabric code, however I have not seen any process related to this job so far.
daygee (Thu, 04 Jan 2018 11:17:17 GMT):
I'm actually looking at the instantiation code right now
daygee (Thu, 04 Jan 2018 11:17:29 GMT):
want to see how it's working
daygee (Thu, 04 Jan 2018 14:08:27 GMT):
User User_1 added by daygee.
daygee (Thu, 04 Jan 2018 14:09:13 GMT):
https://developers.redhat.com/blog/2017/09/22/connecting-kubernetes-docker/
daygee (Thu, 04 Jan 2018 14:09:37 GMT):
a work around
mbuccarello (Fri, 05 Jan 2018 08:09:00 GMT):
Has joined the channel.
mbuccarello (Fri, 05 Jan 2018 11:16:27 GMT):
Hello people, I'm trying to setup fabric under kubernetes
mbuccarello (Fri, 05 Jan 2018 11:17:25 GMT):
what are the best practices to follow in order tu setup fabric in a simple enviroment ( 1 master - 2 worker )
mbuccarello (Fri, 05 Jan 2018 11:17:25 GMT):
what are the best practices to follow in order to setup fabric in a simple enviroment? ( 1 master - 2 worker )
mbuccarello (Fri, 05 Jan 2018 11:20:05 GMT):
I read this think-foundry.com/deploy-hyperledger-fabric-on-kubernetes-part-1/ but I'm not sure if I'm right or not
grapebaba (Sat, 06 Jan 2018 16:14:29 GMT):
any kubernetes cluster is ok according to this guide
JOYELIN (Tue, 09 Jan 2018 08:22:07 GMT):
Has joined the channel.
Taffies (Tue, 09 Jan 2018 08:31:32 GMT):
Has joined the channel.
KathyXu (Tue, 09 Jan 2018 14:22:55 GMT):
Has joined the channel.
mvaibhavshah (Wed, 10 Jan 2018 05:35:27 GMT):
Has joined the channel.
Taffies (Wed, 10 Jan 2018 07:37:48 GMT):
Hello, I'm following Part 2 of the guide on deploying hyper ledger fabric on kubernetes. I tried to start my Fabric cluster but when I viewed my pods, they're all stuck in the state "ContainerCreating". Anyone experienced this before or has an idea what is going on?
Taffies (Wed, 10 Jan 2018 10:24:54 GMT):
Screen Shot 2018-01-10 at 6.23.23 PM.png
milliger (Thu, 11 Jan 2018 14:23:02 GMT):
Has joined the channel.
Luke_Chen (Fri, 12 Jan 2018 03:18:36 GMT):
@Taffies The log shows that it is timed out while trying to connect to your nfs, please check whether your nfs is working as intended.
ArnabChatterjee (Fri, 12 Jan 2018 06:22:54 GMT):
Has joined the channel.
Taffies (Fri, 12 Jan 2018 09:56:40 GMT):
@Luke_Chen I'm not sure what happened, but I'm guessing I mounted something wrongly in my NFS that caused it to go haywire. Anyway, I started from scratch again and able to get the pods running now. :) Thank you!
novusopt (Fri, 12 Jan 2018 14:49:16 GMT):
Has joined the channel.
kipharris (Fri, 12 Jan 2018 17:23:12 GMT):
Has joined the channel.
joaquimpedrooliveira (Fri, 12 Jan 2018 17:47:41 GMT):
Hello, all! I'm facing some chaincode problems when running Fabric on K8S with more than one peer:
I'm starting a network with two orgs, each one with two peers (with their couchdb) and CA, and a solo orderer. I instantiate the chaincode using a peer from Org1 and when I query from Org2 I get:
`Error: Error endorsing query: rpc error: code = Unknown desc = chaincode error (status: 500, message: {"Error":"Nil amount for a"}) -
joaquimpedrooliveira (Fri, 12 Jan 2018 17:50:03 GMT):
And after some time (e.g. 3 hours), without using the network, when I try to query, another error:
`Error: Error endorsing query: rpc error: code = Unknown desc = could not find chaincode with name '12jan1' - make sure the chaincode 12jan1 has been successfully instantiated and try again -
joaquimpedrooliveira (Fri, 12 Jan 2018 17:50:59 GMT):
The commands are being executed on a `fabric-tools` container. When we run the peers using containers over VMs, outside K8S, it works.
joaquimpedrooliveira (Fri, 12 Jan 2018 17:51:07 GMT):
Any tips?
joaquimpedrooliveira (Fri, 12 Jan 2018 18:07:10 GMT):
Pods were not restarted
antoniovassell (Fri, 12 Jan 2018 18:32:51 GMT):
Has joined the channel.
Luke_Chen (Sat, 13 Jan 2018 03:39:44 GMT):
can you get responses when you run `peer node status` on fabric-tools container ?
Luke_Chen (Sat, 13 Jan 2018 03:39:51 GMT):
@joaquimpedrooliveira
Luke_Chen (Sat, 13 Jan 2018 03:41:47 GMT):
could you query or invoke the chaincode after it was instantiated ?
joaquimpedrooliveira (Mon, 15 Jan 2018 14:09:07 GMT):
@Luke_Chen , sorry for the delay. When I run `peer node status` the response is:
```status:STARTED
2018-01-15 14:08:30.240 UTC [main] main -> INFO 001 Exiting.....
```
macsilber (Mon, 15 Jan 2018 23:03:12 GMT):
Has joined the channel.
olrraju (Tue, 16 Jan 2018 01:52:11 GMT):
Has joined the channel.
Taffies (Tue, 16 Jan 2018 08:26:21 GMT):
I'm running the kubernetes network on aws instances and it keeps crashing or stuck in container creating whenever I run the run.py file -- anyone faces similar problems? I did get it to work once as stated above ^ but other times it refuses to run even though I'm following the exact same steps
Taffies (Tue, 16 Jan 2018 08:28:01 GMT):
if anyone is willing to help look into the problem please dm me, I can send you the exact steps I did to recreate the problem :) thank you!
Luke_Chen (Tue, 16 Jan 2018 11:07:17 GMT):
@joaquimpedrooliveira can you find the chaincode container on worker node after the chaincode was instantiated ?
joaquimpedrooliveira (Tue, 16 Jan 2018 14:52:49 GMT):
@Luke_Chen , actually we don't have direct access to nodes due to company policies. I'll ask the team responsible for maintaining the nodes. What's your hypotesis?
joaquimpedrooliveira (Tue, 16 Jan 2018 14:52:49 GMT):
@Luke_Chen , actually we don't have direct access to nodes due to company policies. I'll ask the team responsible for maintaining the nodes. What's your hypothesis?
Luke_Chen (Tue, 16 Jan 2018 15:02:28 GMT):
Because chaincode container will be created by docker daemon of worker node directly, which is not maintained by kubernetes.
Luke_Chen (Tue, 16 Jan 2018 15:03:49 GMT):
make sure your worker nodes have add kube-dns in their DOCKER_OPTS
Luke_Chen (Tue, 16 Jan 2018 15:07:30 GMT):
I think you chaincode wasn't instantiated correctly.
Luke_Chen (Tue, 16 Jan 2018 15:07:30 GMT):
I think the reason that your chaincode wasn't instantiated correctly is the chaincode container failed to start.
joaquimpedrooliveira (Tue, 16 Jan 2018 16:20:12 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=fbpMMG5Pii4zfCbv2) @Luke_Chen I didn't know about this. The settings I made in order to run on kubernetes were the env variables:
``` - name: CORE_VM_DOCKER_ATTACHSTDOUT
value: "true"
- name: CORE_PEER_ADDRESSAUTODETECT
value: "true"
- name: CORE_PEER_TLS_SERVERHOSTOVERRIDE
value: "org-peer1"
```
joaquimpedrooliveira (Tue, 16 Jan 2018 16:20:45 GMT):
I'll check this detail with k8s infra team
joaquimpedrooliveira (Tue, 16 Jan 2018 16:21:26 GMT):
Thank you very much for your help!
ublubu (Tue, 16 Jan 2018 20:20:25 GMT):
Has joined the channel.
wlan0 (Wed, 17 Jan 2018 02:44:34 GMT):
Has joined the channel.
DarrenRatcliffe (Wed, 17 Jan 2018 12:59:05 GMT):
Has joined the channel.
RobertDiebels (Wed, 17 Jan 2018 14:58:48 GMT):
@Luke_Chen I have a question concerning the code you published for BAAS. I took your code and rewrote it to javascript. Would you be willing to assist with any problems I encounter?
RobertDiebels (Wed, 17 Jan 2018 15:07:28 GMT):
My aim with the re-write is to do the following:
- Do not require an NFS-server (Done)
- I fixed this by using ConfigMaps and secrets.
- Fix fabric sym-link issue (Done)
- I wrote a general purpose initContainer which simply copies the any Volume mounted to a path X to path Y. In this case it copies the data from the ConfigMaps/Secrets into a PersitentVolume. This is a work-around until the issue with sym-links is fixed.
- Get all of this working in MiniKube (Semi-Done)
- So far I have PeerOrganizations working. I still need to get OrdererOrganizations functioning but I expect to finish this by the end of the week.
- Get Chaincode running in the cluster (TODO)
- I still need to do this and I expect I will need some help with it. As I've heard that Fabric does some funky things with creating a Docker container for the Chaincode. I read this requires DNS settings, unfortunately I'm not familiar witht his. So I will probably need your help with that.
RobertDiebels (Wed, 17 Jan 2018 15:07:28 GMT):
My aim with the re-write is to do the following:
- Do not require an NFS-server (Done)
- I fixed this by using ConfigMaps and secrets.
- Fix fabric sym-link issue (Done)
-- I wrote a general purpose initContainer which simply copies the any Volume mounted to a path X to path Y. In this case it copies the data from the ConfigMaps/Secrets into a PersitentVolume. This is a work-around until the issue with sym-links is fixed.
- Get all of this working in MiniKube (Semi-Done)
-- So far I have PeerOrganizations working. I still need to get OrdererOrganizations functioning but I expect to finish this by the end of the week.
- Get Chaincode running in the cluster (TODO)
-- I still need to do this and I expect I will need some help with it. As I've heard that Fabric does some funky things with creating a Docker container for the Chaincode. I read this requires DNS settings, unfortunately I'm not familiar witht his. So I will probably need your help with that.
RobertDiebels (Wed, 17 Jan 2018 15:07:28 GMT):
My aim with the re-write is to do the following:
- Do not require an NFS-server (Done)
- I fixed this by using ConfigMaps and secrets.
- Fix fabric sym-link issue (Done)
-- I wrote a general purpose initContainer which simply copies the any Volume mounted to a path X to path Y. In this case it copies the data from the ConfigMaps/Secrets into a PersitentVolume. This is a work-around until the issue with sym-links is fixed.
- Get all of this working in MiniKube (Semi-Done)
-- So far I have PeerOrganizations working. I still need to get OrdererOrganizations functioning but I expect to finish this by the end of the week.
- Get Chaincode running in the cluster (TODO)
-- I still need to do this and I expect I will need some help with it. As I've heard that Fabric does some funky things with creating a Docker container for the Chaincode. I read this requires DNS settings, unfortunately I'm not familiar witht his. So I will probably need your help with that.
RobertDiebels (Wed, 17 Jan 2018 15:07:28 GMT):
My aim with the re-write is to do the following:
- Do not require an NFS-server (Done)
-- I fixed this by using ConfigMaps and secrets.
- Fix fabric sym-link issue (Done)
-- I wrote a general purpose initContainer which simply copies the any Volume mounted to a path X to path Y. In this case it copies the data from the ConfigMaps/Secrets into a PersitentVolume. This is a work-around until the issue with sym-links is fixed.
- Get all of this working in MiniKube (Semi-Done)
-- So far I have PeerOrganizations working. I still need to get OrdererOrganizations functioning but I expect to finish this by the end of the week.
- Get Chaincode running in the cluster (TODO)
-- I still need to do this and I expect I will need some help with it. As I've heard that Fabric does some funky things with creating a Docker container for the Chaincode. I read this requires DNS settings, unfortunately I'm not familiar witht his. So I will probably need your help with that.
RobertDiebels (Wed, 17 Jan 2018 15:07:28 GMT):
My aim with the re-write is to do the following:
- Do not require an NFS-server (Done)
-- I fixed this by using ConfigMaps and secrets.
- Fix fabric sym-link issue (Done)
-- I wrote a general purpose initContainer which simply copies the any Volume mounted to a path X to path Y. In this case it copies the data from the ConfigMaps/Secrets into a PersitentVolume. This is a work-around until the issue with sym-links is fixed. See [this docker hub repo](https://hub.docker.com/r/robertdiebels/funnel/)
- Get all of this working in MiniKube (Semi-Done)
-- So far I have PeerOrganizations working. I still need to get OrdererOrganizations functioning but I expect to finish this by the end of the week.
- Get Chaincode running in the cluster (TODO)
-- I still need to do this and I expect I will need some help with it. As I've heard that Fabric does some funky things with creating a Docker container for the Chaincode. I read this requires DNS settings, unfortunately I'm not familiar witht his. So I will probably need your help with that.
RobertDiebels (Wed, 17 Jan 2018 15:07:28 GMT):
My aim with the re-write is to do the following:
- Do not require an NFS-server (Done)
-- I fixed this by using ConfigMaps and secrets.
- Fix fabric sym-link issue (Done)
-- I wrote a general purpose initContainer which simply copies the any Volume mounted to a path X to path Y. In this case it copies the data from the ConfigMaps/Secrets into a PersistentVolume. This is a work-around until the issue with sym-links is fixed. See [this docker hub repo](https://hub.docker.com/r/robertdiebels/funnel/)
- Get all of this working in MiniKube (Semi-Done)
-- So far I have PeerOrganizations working. I still need to get OrdererOrganizations functioning but I expect to finish this by the end of the week.
- Get Chaincode running in the cluster (TODO)
-- I still need to do this and I expect I will need some help with it. As I've heard that Fabric does some funky things with creating a Docker container for the Chaincode. I read this requires DNS settings, unfortunately I'm not familiar witht his. So I will probably need your help with that.
RobertDiebels (Wed, 17 Jan 2018 15:18:19 GMT):
I don't want to publish my code yet until I've gotten the prototype functioning properly. After that I will clean up the code and publish.
Luke_Chen (Wed, 17 Jan 2018 15:58:07 GMT):
@RobertDiebels Sure, just ping me if you have any problem about networking of chaincode container.
RobertDiebels (Wed, 17 Jan 2018 17:30:18 GMT):
Super thanks :D :thumbsup:
Devender_Singh (Thu, 18 Jan 2018 05:18:45 GMT):
Has joined the channel.
Manish.Sharma (Thu, 18 Jan 2018 09:16:06 GMT):
Has joined the channel.
JayJong (Thu, 18 Jan 2018 10:15:53 GMT):
Hi! I'm having trouble with the very last part of the tutorial with the instantiation of the chaincode. I've added the kube-dns into the docker opts before installing my chaincode.. The installation was successful but there doesn't seem to be any docker container created for the chaincode if it is supposed to be any as well..
JayJong (Thu, 18 Jan 2018 10:16:10 GMT):
Error: Error endorsing chaincode: rpc error: code = Unknown desc = Failed to deserialize creator identity, err MSP Org1MSP is unknown
Katiyman (Thu, 18 Jan 2018 10:57:21 GMT):
Hello All
I raised one issue regarding creation of channel in HLF deployed on kubernetest with custom cryptographic material for MSP
https://stackoverflow.com/questions/48318696/error-while-creating-channel-in-hyperledger-fabric
Kindly help
Luke_Chen (Thu, 18 Jan 2018 12:05:10 GMT):
@JayJong It seems you didn't join peers into any channel
Luke_Chen (Thu, 18 Jan 2018 12:06:54 GMT):
chaincode container will be created after run `peer chaincode instantiate` conmmand
mvaibhavshah (Thu, 18 Jan 2018 13:59:01 GMT):
can you please provide me the link whereby I can find the links to install hyperledger composer, tools , fabric etc in kubernetes?
mvaibhavshah (Thu, 18 Jan 2018 13:59:29 GMT):
I have installed kubernetes on ubuntu and made it master...now want to install hyperledger components
RobertDiebels (Thu, 18 Jan 2018 14:11:53 GMT):
@mvaibhavshah https://labs.vmware.com/flings/blockchain-on-vsphere might be useful.
mrtrantuan (Thu, 18 Jan 2018 15:04:55 GMT):
Has joined the channel.
kenmazsyma (Fri, 19 Jan 2018 01:53:38 GMT):
Has joined the channel.
JayJong (Fri, 19 Jan 2018 03:37:32 GMT):
@Luke_Chen ok i managed to get it working, thanks!
Katiyman (Fri, 19 Jan 2018 06:09:40 GMT):
Hello All
I raised one issue regarding creation of channel in HLF deployed on kubernetest with custom cryptographic material for MSP
https://stackoverflow.com/questions/48318696/error-while-creating-channel-in-hyperledger-fabric
Kindly help
cotofei (Fri, 19 Jan 2018 06:32:38 GMT):
Has joined the channel.
vanitas92 (Fri, 19 Jan 2018 10:55:59 GMT):
Has joined the channel.
vanitas92 (Fri, 19 Jan 2018 10:58:30 GMT):
Hi guys do you know if there is any plans on supporting kubernetes for the chaincode instantiation? Its a massive con in our organization as they were extremely interested in implementing hyperledger in kubernetes.
joaquimpedrooliveira (Fri, 19 Jan 2018 13:44:46 GMT):
@vanitas92 , I think some work related to this is being done in https://jira.hyperledger.org/browse/FAB-7406
Brucepark (Sat, 20 Jan 2018 06:15:04 GMT):
Has joined the channel.
tupt (Mon, 22 Jan 2018 03:22:55 GMT):
Has joined the channel.
tupt (Mon, 22 Jan 2018 04:38:11 GMT):
Hi all, I am setting up hyperledger fabric with kubernetes using minikube as worker node, but though after changing docker dns seems chaincode container can not be deployed, my docker config file is {
"dns" : [
"10.96.0.10",
"192.168.99.100"
],
"dns-search" : [
"org1-f-1.svc.cluster.local",
"org2-f-1.svc.cluster.local",
"orgorderer-f-1.svc.cluster.local",
"default.svc.cluster.local",
"svc.cluster.local"
]
}
Taffies (Tue, 23 Jan 2018 05:52:14 GMT):
Hello! If I understand correctly, communications between the worker nodes and the master node on the kubernetes network are done through the API server, so basically the master node sort of controls all the activity going on in the network? But if I want to send up my kubernetes network on multiple worker nodes (e.g. one VM for each organization), would it be possible to invoke a transaction through the worker nodes themselves, or do I need to go through the master node no matter what?
Glen (Tue, 23 Jan 2018 07:06:33 GMT):
Has joined the channel.
tupt (Tue, 23 Jan 2018 07:45:21 GMT):
Have anyone tried setup hyperledger fabric with minikube on local one successfully?
Taffies (Tue, 23 Jan 2018 08:45:05 GMT):
Hello! If I understand correctly, all communication between the master & worker node is done via the apiserver & kubelet. Right now, I'm invoking transactions from within the CLI pod of each org from my master node. However, I want to set up my network such that there is one master node and one worker node for the pods of each organization that I'm creating, and be able to invoke transactions from outside the master node, specifically from a webapp.
I've got a few questions regarding this:
1. Do you all have have any suggestions on the best way to go about doing this? (I've tried to find tutorials on how to invoke externally but many of the pages doesn't really go in-depth into this topic)
2. My current set-up is based on the fact that I want each org to be able to own their own node.. Kinda like decentralising the network & for redundancy purposes? Do you think this is a good idea, or is it redundant?
3. In Kubernetes, if all requests go through the APIserver on the master node.. doesn't it make the entire network centralised?
PS: Relatively new to all of this so I'm dealing with quite a lot of confusion on my part. Sorry if I've got any concepts wrong! :P
C0rWin (Tue, 23 Jan 2018 14:22:45 GMT):
Has joined the channel.
julian (Tue, 23 Jan 2018 17:31:05 GMT):
Has joined the channel.
Luke_Chen (Wed, 24 Jan 2018 02:37:41 GMT):
for invoking transactions from external, you should define a nodePort type service for pods, mapping ports of pod to ports of node(worker)
Luke_Chen (Wed, 24 Jan 2018 02:39:06 GMT):
and then access service that running in pod by address worker_ip:port
Luke_Chen (Wed, 24 Jan 2018 02:42:37 GMT):
due to the network proxy of kubenetes, you can use any node_ip as worker_ip in your cluster.
Luke_Chen (Wed, 24 Jan 2018 02:43:00 GMT):
node_ip = ip of node :)
Luke_Chen (Wed, 24 Jan 2018 02:46:52 GMT):
meanwhile, I don't think kubenetes can do this, it schedules pods according to load balance `2. My current set-up is based on the fact that I want each org to be able to own their own node.. Kinda like decentralising the network & for redundancy purposes? Do you think this is a good idea, or is it redundant? `
Luke_Chen (Wed, 24 Jan 2018 02:46:52 GMT):
meanwhile, I don't think kubenetes can do this, it schedules pods according to load balance `2. My current set-up is based on the fact that I want each org to be able to own their own node.. Kinda like decentralising the network & for redundancy purposes? Do you think this is a good idea, or is it redundant? `
Luke_Chen (Wed, 24 Jan 2018 02:46:52 GMT):
meanwhile, I don't think kubenetes can do this, it schedules pods according to load balance
`2. My current set-up is based on the fact that I want each org to be able to own their own node.. Kinda like decentralising the network & for redundancy purposes? Do you think this is a good idea, or is it redundant? `
Luke_Chen (Wed, 24 Jan 2018 02:46:52 GMT):
meanwhile, I don't think it is a good idea, kubernetes schedule pods according to load balance, better not break this balance
`2. My current set-up is based on the fact that I want each org to be able to own their own node.. Kinda like decentralising the network & for redundancy purposes? Do you think this is a good idea, or is it redundant? `
tupt (Wed, 24 Jan 2018 03:29:37 GMT):
can we use custom dns to setup chaincode instantiation for hyperledger?
tupt (Wed, 24 Jan 2018 03:31:27 GMT):
http://blog.kubernetes.io/2017/04/configuring-private-dns-zones-upstream-nameservers-kubernetes.html
Taffies (Wed, 24 Jan 2018 04:45:10 GMT):
@Luke_Chen Ok thanks so much! I'll play around with the nodePort type service and see if I can externally invoke it successfully. One quick clarification - in the tutorial, it states that the port of the ordererN is defined as 33700 + N, but it's defined as 32000 + N in the config files. Is this a typo?
Luke_Chen (Wed, 24 Jan 2018 04:45:38 GMT):
yes~ sorry for that
Taffies (Wed, 24 Jan 2018 04:50:41 GMT):
Ok thanks for the clarification! And thank you for your input about the scheduling of pods :)
Taffies (Wed, 24 Jan 2018 04:51:08 GMT):
@Luke_Chen Ok thanks for the clarification! And thank you for your input about the scheduling of pods 🙂
Katiyman (Wed, 24 Jan 2018 06:42:11 GMT):
HI
requesting if someone can help me with below issue
https://stackoverflow.com/questions/48318696/error-while-creating-channel-in-hyperledger-fabric
Luke_Chen (Wed, 24 Jan 2018 08:12:14 GMT):
@Katiyman I am interesting in your issue, but have no idea so far.
Katiyman (Wed, 24 Jan 2018 08:15:00 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=6qo2NWMye3vn2nRfZ) @Luke_Chen I removed the custom(3rd party) certificates
Katiyman (Wed, 24 Jan 2018 08:15:00 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=6qo2NWMye3vn2nRfZ) @Luke_Chen I removed the custom(3rd party) certificates. but still getting issue
Katiyman (Wed, 24 Jan 2018 08:15:00 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=6qo2NWMye3vn2nRfZ) @Luke_Chen I removed the custom(3rd party) certificates.And used cryptogen for cerificates. but still getting issue.
Luke_Chen (Wed, 24 Jan 2018 13:36:27 GMT):
Have you try to redeploy the clusters?
Luke_Chen (Wed, 24 Jan 2018 13:37:30 GMT):
Which version of Fabric docker images you are using?
Luke_Chen (Wed, 24 Jan 2018 13:37:30 GMT):
Which version of Fabric docker images are you using?
Luke_Chen (Wed, 24 Jan 2018 13:37:30 GMT):
@Katiyman Which version of Fabric docker images are you using?
RobertDiebels (Wed, 24 Jan 2018 13:48:18 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=ngtGnu42LxAYKz5px) @tupt I have done so successfully.
tupt (Wed, 24 Jan 2018 15:35:35 GMT):
can you help me
tupt (Wed, 24 Jan 2018 15:35:51 GMT):
I am stuck at instantiating chaincode step @RobertDiebels
indira.kalagara (Thu, 25 Jan 2018 06:44:03 GMT):
Has joined the channel.
tupt (Thu, 25 Jan 2018 09:27:45 GMT):
https://hackernoon.com/how-to-deploy-hyperledger-fabric-on-kubernetes-1-a2ceb3ada078 how to set --dns
RobertDiebels (Thu, 25 Jan 2018 10:07:34 GMT):
@tupt I haven't fiddled with the DNS step nor am I doing anything with NFS. I'm creating a setup so that you don't have to touch the DNS anymore. So I wouldn't know how to handle the DNS.
antoniovassell (Thu, 25 Jan 2018 19:52:44 GMT):
@RobertDiebels how are you going about doing that? [ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=AtGGgg4BRphqRuXxz) @RobertDiebels
JayJong (Fri, 26 Jan 2018 08:24:42 GMT):
@Luke_Chen Hi, in the file fabric_1_0_template_pod_cli.yaml, is there a reason why u put value of CORE_PEER_TLS_ENABLED as false? and why are the following env variables(CORE_PEER_TLS_CERT_FILE, CORE_PEER_TLS_KEY_FILE, CORE_PEER_TLS_ROOTCERT_FILE) commented out?
niyuelin (Fri, 26 Jan 2018 08:50:28 GMT):
Has joined the channel.
Luke_Chen (Fri, 26 Jan 2018 14:06:43 GMT):
@JayJong We have not tested enabled tls yet, to avoid unpredictable behaviour we commented those variables out.
Luke_Chen (Fri, 26 Jan 2018 14:06:43 GMT):
@JayJong We have not tested enabled tls yet, we commented those variables out to avoid unpredictable behaviour .
RobertDiebels (Fri, 26 Jan 2018 20:21:42 GMT):
@antoniovassell I followed a lot of the work by @Luke_Chen . With some minor exceptions. First, I've build some code to transform an Organizations' directory into ConfigMaps and Secrets (where applicable). Second, I'm also creating a bit of code which will generate a Job to: 1. create a Channel, 2. Join x amount of peers to said channel 3. Install chaincode on those peers. 4. (this is the dirty part) listen to the hostPaths unix socket for docker then a. If an image is created via the docker socket, pick up on that and push the image to a repository y. 5. Then launch a deployment which uses that image to run a container.
RobertDiebels (Fri, 26 Jan 2018 20:23:15 GMT):
Now this is just the theory of course. I'm bound to bump into some roadblocks. I'm about to test steps 1 through 3 to see if that works.
RobertDiebels (Fri, 26 Jan 2018 20:24:16 GMT):
I want to release a pre-alpha soon. So people can have a go at it. So far I've only tested on Minikube. But I'm working on some ideas so that users can define custom targets for the cluster.
tupt (Sat, 27 Jan 2018 07:55:32 GMT):
I did it using kubernetes api, first instantiate with 1 second timeout to make sure image created, then redo it while peer is waiting for container to connect, then it worked
RobertDiebels (Sat, 27 Jan 2018 09:26:24 GMT):
@tupt Yes but that would require user intiated interaction right? I'm writing some code to do this instead of the user.
tupt (Sat, 27 Jan 2018 13:47:56 GMT):
very nice, it is automation using kubeapi, I used api pod with node sdk to do that
RobertDiebels (Sat, 27 Jan 2018 19:29:30 GMT):
@tupt any chance you could post that code somewhere?
tupt (Sun, 28 Jan 2018 02:00:50 GMT):
done.png
tupt (Sun, 28 Jan 2018 02:05:36 GMT):
kubectl run chaincode --image=$(docker images --format "{{.ID}} {{.Name}}" | grep chaincode.version | awk '{print $1}') && kubectl exec -it chaincode -- CORE_CHAINCODE_ID_NAME=mycc:1 --peer.address=peer0.org1:7051
tupt (Sun, 28 Jan 2018 02:06:08 GMT):
but my suggestion is using kube-api inside fabric-client sdk to do that
tupt (Sun, 28 Jan 2018 02:07:03 GMT):
sendInstantiateProposal => kube-api create pod from image => .then(results => {})
RobertDiebels (Sun, 28 Jan 2018 09:36:59 GMT):
Thanks :D
RobertDiebels (Sun, 28 Jan 2018 09:37:13 GMT):
I'll have a go at it.
Luke_Chen (Sun, 28 Jan 2018 12:58:53 GMT):
@tupt Does fabric-client sdk have kube-api inside?
RobertDiebels (Sun, 28 Jan 2018 15:59:22 GMT):
@Luke_Chen It doesn't. I'm using the godaddy kubernetes-client to do the things I need. https://github.com/godaddy/kubernetes-client
fengfengs (Mon, 29 Jan 2018 05:05:56 GMT):
Has joined the channel.
kipharris (Mon, 29 Jan 2018 15:10:11 GMT):
"IBM Cloud Private"
tupt (Mon, 29 Jan 2018 16:55:20 GMT):
Here is my project using that strategy, I have did it using kuber-api but this is my secret project. https://github.com/tubackkhoa/hyperledger-kubernetes/tree/develop
tupt (Mon, 29 Jan 2018 16:56:03 GMT):
process.png
Luke_Chen (Tue, 30 Jan 2018 03:30:37 GMT):
@tupt It seems good !
cmgabriel (Wed, 31 Jan 2018 21:14:50 GMT):
Has joined the channel.
vchengsong (Thu, 01 Feb 2018 03:49:34 GMT):
Has joined the channel.
oleg.borovyk (Thu, 01 Feb 2018 17:29:45 GMT):
Has joined the channel.
berestet (Thu, 01 Feb 2018 17:42:32 GMT):
Has joined the channel.
GirijaShankarMishra (Sat, 03 Feb 2018 10:27:39 GMT):
Has joined the channel.
GirijaShankarMishra (Sat, 03 Feb 2018 10:29:35 GMT):
hyperledger.png
GirijaShankarMishra (Sat, 03 Feb 2018 10:37:43 GMT):
I have followed Deploy hyperledger fabric on kubernetes part 2
mamtabhardwaj12 (Sat, 03 Feb 2018 11:07:18 GMT):
Has joined the channel.
mamtabhardwaj12 (Sat, 03 Feb 2018 11:10:01 GMT):
hyperledger.png
Luke_Chen (Sun, 04 Feb 2018 03:45:52 GMT):
@GirijaShankarMishra It seems you can not connect to your orderer
Luke_Chen (Sun, 04 Feb 2018 03:45:52 GMT):
@GirijaShankarMishra It seems can not connect to your orderer
Luke_Chen (Sun, 04 Feb 2018 03:47:37 GMT):
can you check wehther your orderer is running or not
Luke_Chen (Sun, 04 Feb 2018 03:48:12 GMT):
if it was running, please check it's log
indira.kalagara (Mon, 05 Feb 2018 07:10:47 GMT):
Hi team,
I have set up the Blockchain network by following the simple install instructions from https://ibm-blockchain.github.io/simple/
Everything works fine and I could use composer playground , deploy business network and test and perform transactions.
However after a week or later, the same business network connection is failing with error:
0|composer | at /home/composer/.npm-global/lib/node_modules/@ibmblockchain/composer-playground/node_modules/grpc/src/node/src/client.js:554:15
0|composer | HLFConnection :queryChainCode() Error: Error trying to query business network. Error: Failed to deserialize creator identity, err MSP Org1MSP is unknown
Then I have observed that Peer1 is not listing the channel "channel1" which it joined earlier. "peer channel list" command from peer1 / org1 is not listing channel.
Where as the same command from Org2 Peer1 list the channlel - "channel1".
We haven't peformed any operations on kubernetes cluster / blockchain apart from accessing Composer Playground.
As per my understanding the unknown MSP ID error is thrown because the peer is not joined the channel. But I am not sure the reason why Org1 Peer1 is not part of the channel anymore
where it joined the same channel initially and we were able to deploy bna and invoke, etc.
What is causing the peer to not part of the channel ? Is it becuase of some pod restart (not sure if it happened ) ?
This behaviour has been observed couple of times so far and we are recreating the n/w everytime.
Please provide some inputs if you come across this scenario and what steps we should verify and take care to handle this behavior.
Thanks,
Indira
JayJong (Mon, 05 Feb 2018 09:01:35 GMT):
Hi, I got this error when i invoke using the invoke.js file in fabcar.
```
Failed to invoke successfully :: Error: There was a problem with the eventhub ::Error: 14 UNAVAILABLE: Connect Failed
```
The default port for the event hub in the fabcar example is 7053. I am currently deploying my fabric on kubernetes. My peer0org1 maps 7051:30001 and 7052:30002. I tried setting to either of these ports but it still didnt work. What should i set my port to?
mamtabhardwaj12 (Mon, 05 Feb 2018 09:11:21 GMT):
hyperledger2.png
mamtabhardwaj12 (Mon, 05 Feb 2018 09:11:21 GMT):
hyperledger2.png
mamtabhardwaj12 (Mon, 05 Feb 2018 09:25:48 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=P7HcXyRAwasY5gb9N) @tupt Have you done your create channel successfully? I am not able to do create a channel. Can you please help me out?
Luke_Chen (Mon, 05 Feb 2018 11:08:10 GMT):
@mamtabhardwaj12 could you please show me orderer logs ?
mamtabhardwaj12 (Mon, 05 Feb 2018 11:24:20 GMT):
hyperledger2.png
mamtabhardwaj12 (Mon, 05 Feb 2018 11:24:20 GMT):
hyperledger2.png
Luke_Chen (Mon, 05 Feb 2018 11:25:03 GMT):
I mean logs , container logs
mamtabhardwaj12 (Mon, 05 Feb 2018 11:26:06 GMT):
wait
mamtabhardwaj12 (Mon, 05 Feb 2018 11:52:00 GMT):
hyperledger_logs1.png
mamtabhardwaj12 (Mon, 05 Feb 2018 11:52:14 GMT):
hyperledger_logs2.png
mamtabhardwaj12 (Mon, 05 Feb 2018 11:53:26 GMT):
hyperledger_logs3.png
mamtabhardwaj12 (Mon, 05 Feb 2018 11:53:55 GMT):
hyperledger_logs4.png
mamtabhardwaj12 (Mon, 05 Feb 2018 11:54:14 GMT):
hyperledger_logs5.png
mamtabhardwaj12 (Mon, 05 Feb 2018 11:54:40 GMT):
hyperledger_logs6.png
mamtabhardwaj12 (Mon, 05 Feb 2018 11:55:26 GMT):
hyperledger_logs7.png
mamtabhardwaj12 (Mon, 05 Feb 2018 11:55:59 GMT):
hyperledger_logs8.png
mamtabhardwaj12 (Mon, 05 Feb 2018 11:56:17 GMT):
@Luke_Chen Can you please look into it??
Luke_Chen (Mon, 05 Feb 2018 11:57:58 GMT):
orderer seems normal, can you run peer node status in cli container ?
mamtabhardwaj12 (Mon, 05 Feb 2018 12:02:01 GMT):
I
mamtabhardwaj12 (Mon, 05 Feb 2018 12:02:11 GMT):
Peers are running
Luke_Chen (Mon, 05 Feb 2018 12:02:26 GMT):
orderer seems normal, can you run `peer node status` in cli container ?
mamtabhardwaj12 (Mon, 05 Feb 2018 12:03:11 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=eky3tJqFoaGRSpJ4h) @Luke_Chen So you want me to run this cammand inside cli org1 container?
Luke_Chen (Mon, 05 Feb 2018 12:03:13 GMT):
I konw they are running, but it may have some problem of your network
Luke_Chen (Mon, 05 Feb 2018 12:03:19 GMT):
yes
mamtabhardwaj12 (Mon, 05 Feb 2018 12:03:23 GMT):
ok
mamtabhardwaj12 (Mon, 05 Feb 2018 12:07:46 GMT):
peer-status.png
mamtabhardwaj12 (Mon, 05 Feb 2018 12:07:46 GMT):
peer-status.png
Luke_Chen (Mon, 05 Feb 2018 12:34:50 GMT):
@mamtabhardwaj12 some thing is wrong with your kubernetes network
Luke_Chen (Mon, 05 Feb 2018 12:35:02 GMT):
How did you setup kubernetes network?
Luke_Chen (Mon, 05 Feb 2018 12:35:06 GMT):
by kubeadm?
mamtabhardwaj12 (Mon, 05 Feb 2018 12:35:09 GMT):
ya
mamtabhardwaj12 (Mon, 05 Feb 2018 12:35:36 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=iLiNKy2GRTB76pX89) @Luke_Chen by kubeadm
mamtabhardwaj12 (Mon, 05 Feb 2018 12:37:15 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=iLiNKy2GRTB76pX89) @Luke_Chen Is there problem of firewall? because we have disable it. When we are enable firewall then nfs client not able to connect.
mamtabhardwaj12 (Mon, 05 Feb 2018 12:38:53 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=iLiNKy2GRTB76pX89) @Luke_Chen I have followed this document for creating cluster https://www.mirantis.com/blog/how-install-kubernetes-kubeadm/
Luke_Chen (Mon, 05 Feb 2018 12:39:00 GMT):
@mamtabhardwaj12 properly, try to disable iptable first
mamtabhardwaj12 (Mon, 05 Feb 2018 12:39:34 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=gDRXCqADRXpBKrhcG) @Luke_Chen Is this a reason?
Luke_Chen (Mon, 05 Feb 2018 12:42:16 GMT):
could be one
mamtabhardwaj12 (Mon, 05 Feb 2018 12:43:12 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=5BXadmdWddBrSP4qh) @Luke_Chen just please tell me one thing, currently my firewall is disabled.So do I need to disable iptables?
Luke_Chen (Mon, 05 Feb 2018 12:43:26 GMT):
Noop
mamtabhardwaj12 (Mon, 05 Feb 2018 12:43:43 GMT):
no n?
mamtabhardwaj12 (Mon, 05 Feb 2018 12:44:07 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=5DiKbeXQAyRsqK7hG) @Luke_Chen no right?
Luke_Chen (Mon, 05 Feb 2018 12:45:20 GMT):
let me go through the link you pasted first,
mamtabhardwaj12 (Mon, 05 Feb 2018 12:45:31 GMT):
ya, please
mamtabhardwaj12 (Mon, 05 Feb 2018 12:46:46 GMT):
I have also followed this link for NFS https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nfs-mount-on-ubuntu-16-04
Luke_Chen (Mon, 05 Feb 2018 12:50:08 GMT):
@mamtabhardwaj12 could paste the result of `kubectl get pods --all-namespaces` here ?
mamtabhardwaj12 (Mon, 05 Feb 2018 12:57:54 GMT):
all_pods.png
mamtabhardwaj12 (Mon, 05 Feb 2018 12:57:54 GMT):
all_pods.png
Luke_Chen (Mon, 05 Feb 2018 13:06:30 GMT):
@mamtabhardwaj12 that's weird
Luke_Chen (Mon, 05 Feb 2018 13:06:52 GMT):
It seems everything works
mamtabhardwaj12 (Mon, 05 Feb 2018 13:07:02 GMT):
yup
Luke_Chen (Mon, 05 Feb 2018 13:07:11 GMT):
how about change another add-on network?
Luke_Chen (Mon, 05 Feb 2018 13:07:17 GMT):
like flannel
mamtabhardwaj12 (Mon, 05 Feb 2018 13:11:06 GMT):
Calico
mamtabhardwaj12 (Mon, 05 Feb 2018 13:11:06 GMT):
Calico @Luke_Chen
Luke_Chen (Mon, 05 Feb 2018 13:12:43 GMT):
you are using calico
mamtabhardwaj12 (Mon, 05 Feb 2018 13:12:59 GMT):
ya @Luke_Chen
Luke_Chen (Mon, 05 Feb 2018 13:13:37 GMT):
I mean you use another add-on network an try all over again
Luke_Chen (Mon, 05 Feb 2018 13:13:37 GMT):
I mean you use another add-on network and try all over again
mamtabhardwaj12 (Mon, 05 Feb 2018 13:13:58 GMT):
ok @Luke_Chen
mamtabhardwaj12 (Mon, 05 Feb 2018 13:17:03 GMT):
Thank you so much ... will let you know @Luke_Chen
Luke_Chen (Mon, 05 Feb 2018 13:17:43 GMT):
@mamtabhardwaj12 ok
antoniovassell (Mon, 05 Feb 2018 20:39:07 GMT):
Hey, was anyone able to get kafka working with the orderer correctly on Kubernetes?
pasimoes (Mon, 05 Feb 2018 21:50:07 GMT):
Has joined the channel.
mamtabhardwaj12 (Tue, 06 Feb 2018 09:49:19 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=yh7pGp2dqDtqeE2pS) @Luke_Chen Hey, I have tried with new add-on network as flannel but error is same while creating channel.
ruchika05 (Tue, 06 Feb 2018 09:57:01 GMT):
Has joined the channel.
mamtabhardwaj12 (Tue, 06 Feb 2018 10:34:00 GMT):
hyperledger error.png
mamtabhardwaj12 (Tue, 06 Feb 2018 10:38:08 GMT):
hyperledger error.png
mamtabhardwaj12 (Tue, 06 Feb 2018 10:38:08 GMT):
hyperledger error.png
mamtabhardwaj12 (Tue, 06 Feb 2018 10:38:08 GMT):
hyperledger error.png
Luke_Chen (Tue, 06 Feb 2018 11:31:35 GMT):
@mamtabhardwaj12 It must be network wasn't work as intended
Luke_Chen (Tue, 06 Feb 2018 11:31:35 GMT):
@mamtabhardwaj12 It must be network doesn't work as intended
mamtabhardwaj12 (Tue, 06 Feb 2018 11:39:56 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=QMEjtQG9CgaJwcSE7) @Luke_Chen So what will be solution??
mamtabhardwaj12 (Tue, 06 Feb 2018 11:39:56 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=QMEjtQG9CgaJwcSE7) @Luke_Chen So what will be solution?? do you have any idea?
Luke_Chen (Tue, 06 Feb 2018 14:08:10 GMT):
@mamtabhardwaj12 Oh man
Luke_Chen (Tue, 06 Feb 2018 14:08:24 GMT):
you can create channel now
Luke_Chen (Tue, 06 Feb 2018 14:09:03 GMT):
I saw you get response from `peer node status`
Luke_Chen (Tue, 06 Feb 2018 14:10:27 GMT):
There is a typo in you channel creation command, it should be orderer0.orgorderer1:7050 rather than orderer0.orgorderer1: 7050
Luke_Chen (Tue, 06 Feb 2018 14:11:46 GMT):
apology for my careless, your network is totaly fine
Luke_Chen (Tue, 06 Feb 2018 14:11:46 GMT):
apology for my careless, your network is totally fine right now.
vanitas92 (Tue, 06 Feb 2018 14:15:12 GMT):
Hi everyone, i'm trying to setup a fabric network with kafka cluster, everyting in the same kubernetes cluster. I have deployed kafka previously as 4 brokers, each one as a separate deployment, now im tryng to deploy it using statefulsets. Now i'm having this issue when setting up the orderer. It crashes when it is trying to connect to the kafka cluster. I'm exposing the kafka cluster as a headless service at port 9092 and in the configtx.yaml is declaring a unique broker pointing at the headless service. The orderer crashes when trying to get an IP. I think it is something of the headless service and maybe it is not supported. I'm posting the output of the crash.
vanitas92 (Tue, 06 Feb 2018 14:17:56 GMT):
Captura.PNG
vanitas92 (Tue, 06 Feb 2018 14:18:52 GMT):
Captura2.PNG
vanitas92 (Tue, 06 Feb 2018 14:20:11 GMT):
Captura3.PNG
Luke_Chen (Tue, 06 Feb 2018 14:22:44 GMT):
@vanitas92 Which could provider are you using?
Luke_Chen (Tue, 06 Feb 2018 14:22:58 GMT):
or they are local kubernetes cluster?
vanitas92 (Tue, 06 Feb 2018 14:28:29 GMT):
@Luke_Chen Everything (the orderer and the kafka cluster) are in the same local kubernetes cluster and in the same namespace
vanitas92 (Tue, 06 Feb 2018 14:29:01 GMT):
@Luke_Chen im using calico as the network plugin
Luke_Chen (Tue, 06 Feb 2018 14:29:53 GMT):
can it work in solo mode？
vanitas92 (Tue, 06 Feb 2018 14:31:50 GMT):
@Luke_Chen yes it work as solo mode and also in kafka mode but as the cluster set up as 4 different deployments. Now im trying to recreate the kafka cluster as statefulset with 4 replicas
vanitas92 (Tue, 06 Feb 2018 14:33:28 GMT):
@Luke_Chen if you look at the output images, the ordere is resolving correctly the DNS of the kafka cluster, first pointing at the headless service endpoint, and then discovering the 4 brokers
Luke_Chen (Tue, 06 Feb 2018 14:34:16 GMT):
yes, I saw that
Luke_Chen (Tue, 06 Feb 2018 14:35:10 GMT):
Clipboard - 2018年2月6日晚上10点35分
Luke_Chen (Tue, 06 Feb 2018 14:36:55 GMT):
this is a known problem occurring in some cloud provider,
Luke_Chen (Tue, 06 Feb 2018 14:36:55 GMT):
this is a known problem will occur in some cloud provider.
Luke_Chen (Tue, 06 Feb 2018 14:38:20 GMT):
maybe try to add this argument in orderer and kafka brokers deployment file
Luke_Chen (Tue, 06 Feb 2018 14:38:20 GMT):
maybe try to add this argument to orderer and kafka brokers deployment file
Luke_Chen (Tue, 06 Feb 2018 14:38:20 GMT):
maybe try to add this environment to orderer and kafka brokers deployment file
Luke_Chen (Tue, 06 Feb 2018 14:38:20 GMT):
try to add this environment to orderer and kafka brokers deployment file
Luke_Chen (Tue, 06 Feb 2018 14:39:44 GMT):
Key:GODEBUG value:netdns=go
vanitas92 (Tue, 06 Feb 2018 15:36:16 GMT):
okay thanks ill try it later and ill post the result
malengatiger (Tue, 06 Feb 2018 18:59:22 GMT):
Has joined the channel.
mamtabhardwaj12 (Wed, 07 Feb 2018 07:59:21 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=KxAvg9S6dzbpctRJZ) @Luke_Chen I have tried it earlier by removing space from port but still, error is same.
Luke_Chen (Wed, 07 Feb 2018 08:12:17 GMT):
I saw you get correct response from `peer node status`
Luke_Chen (Wed, 07 Feb 2018 08:14:00 GMT):
the error is same as connection unavailable?
mamtabhardwaj12 (Wed, 07 Feb 2018 08:40:13 GMT):
ya @Luke_Chen
Luke_Chen (Wed, 07 Feb 2018 08:47:12 GMT):
@mamtabhardwaj12 I gussed your dns is still keep restarting ?
Luke_Chen (Wed, 07 Feb 2018 08:47:12 GMT):
@mamtabhardwaj12 I guessed your dns is still keep restarting ?
Luke_Chen (Wed, 07 Feb 2018 08:47:12 GMT):
@mamtabhardwaj12 I guessed your dns still keep restarting ?
mamtabhardwaj12 (Wed, 07 Feb 2018 08:50:38 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=hNEnd22pJ2jco3x95) @Luke_Chen how I can check it?
Luke_Chen (Wed, 07 Feb 2018 08:50:38 GMT):
could you paste your dns' logs here ?
mamtabhardwaj12 (Wed, 07 Feb 2018 08:52:29 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=f6srWYcaABeBuYM4R) @Luke_Chen Can you please give me command? if possible
Luke_Chen (Wed, 07 Feb 2018 08:56:25 GMT):
`kubectl logs --namespace=kube-system kube-dns-xxxxx
Luke_Chen (Wed, 07 Feb 2018 08:56:25 GMT):
`kubectl logs --namespace=kube-system kube-dns-xxxxx`
Luke_Chen (Wed, 07 Feb 2018 08:57:33 GMT):
or `kubectl get pods --all-namespaces`
mamtabhardwaj12 (Wed, 07 Feb 2018 09:10:24 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=KtJ9CR9eeFZqXXb5M) @Luke_Chen Hey, I am not able to do it right now, Actually, I have some urgent work.Will let you know about progress.Can you please give me your email id, I have searched it on the published document but not able to make it.
vanitas92 (Wed, 07 Feb 2018 11:13:50 GMT):
@Luke_Chen I have inserted the variable that you suggested and it is not debugging anything related to netdns. However, without any extra change, i have redeployed kafka and orderer and its working now. Does this bug happen arbitrarily?
Luke_Chen (Wed, 07 Feb 2018 11:56:15 GMT):
@vanitas92 yes, it does
vanitas92 (Wed, 07 Feb 2018 12:03:33 GMT):
@Luke_Chen Ok thank you! I'll rollback to the 4-broker kafka cluster using deployments instead of statefulset!
AshishMishra 1 (Wed, 07 Feb 2018 12:04:21 GMT):
Has joined the channel.
RobertDiebels (Wed, 07 Feb 2018 16:40:13 GMT):
Hey guys, for those who are interested. I'm planning on publishing my Fabric Kubernetes project some time next week. Both on Github and on NPM.
RobertDiebels (Wed, 07 Feb 2018 16:40:44 GMT):
I'm hoping people on Linux can test it out for me (Windows user here).
RobertDiebels (Wed, 07 Feb 2018 16:41:38 GMT):
Essentially it's a CLI that allows you to create Kuberenetes configuration from Blockchain configuration.
RobertDiebels (Wed, 07 Feb 2018 16:42:38 GMT):
Source is written in ES6 and it runs on Node.js.
RobertDiebels (Wed, 07 Feb 2018 16:44:18 GMT):
For now it only supports Minikube as a deployment target however I'm working on some code to allow any target the user wants.
RobertDiebels (Wed, 07 Feb 2018 16:44:43 GMT):
Will post again when I've published.
mastersingh24 (Wed, 07 Feb 2018 18:06:41 GMT):
Has joined the channel.
AshishMishra 1 (Thu, 08 Feb 2018 06:27:00 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=heSdCXNm3BWGfoWDH) @RobertDiebels
Nice, I'd like to participate. I 've working setups with Swarm but I feel k8 would give more benefits.
RobertDiebels (Thu, 08 Feb 2018 12:06:20 GMT):
That would be great :D
Luke_Chen (Thu, 08 Feb 2018 13:50:18 GMT):
@RobertDiebels wow, can't wait to see your project;)
Luke_Chen (Thu, 08 Feb 2018 13:50:18 GMT):
@RobertDiebels wow, can't wait to see your project ;)
RobertDiebels (Thu, 08 Feb 2018 13:51:05 GMT):
Writing the README now :P
krabradosty (Thu, 08 Feb 2018 17:03:36 GMT):
Has joined the channel.
tsnyder (Fri, 09 Feb 2018 18:15:04 GMT):
I just uploaded a java sdk wrapper with a Fabric kubernetes minikube for local development. It is located at https:github.com/t-snyder/fabric-dev . Contents are Docker and Kubernetes deployments for 4 Fabric Peers, a solo Orderer, CLI, a Kafka / Zookeeper cluster for messaging in/out, and an sdk-wrapper service for each org. Its purpose is more around being able to set up a local dev environment. I appreciate all feedback. The doc is located in the setup-dev/doc directory.
nycycr (Sat, 10 Feb 2018 06:27:47 GMT):
Has joined the channel.
ublubu (Mon, 12 Feb 2018 02:16:49 GMT):
`peer channel join -b channel.block` -> `failed to create new connection: x509: certificate is valid for peer0.org1, peer0, not peer0-org1` -- how do you all deal with the discrepancy between the standard `
ublubu (Mon, 12 Feb 2018 05:14:57 GMT):
specifically, @tsnyder, i'm looking at https://github.com/t-snyder/fabric-dev/blob/8548ef08bed94ae378b57c10ff803e1c210fd0f0/setup-dev/kube/peer0.yaml#L152 , where the `peer0.Org1` service is called `peer0-org1`.
i found that `peer channel join -b channel.block` works if i name the peer's service `peer0` instead of `peer0-org1`, but if `Org2` also has a `peer0`, i can't name both peers `peer0`.
ublubu (Mon, 12 Feb 2018 06:05:15 GMT):
also, has anyone tried the code here: https://jira.hyperledger.org/browse/FAB-7406 https://github.com/estaleiro/fabric/commits/issue-7406
vanitas92 (Mon, 12 Feb 2018 14:10:31 GMT):
@tupt Hi, would you please pass the yaml file that you used to deploy the chaincode instantiation in kubernetes? The link that you paste here (https://github.com/tubackkhoa/hyperledger-kubernetes/tree/develop) is no longer working. Thanks!!
ublubu (Mon, 12 Feb 2018 17:52:36 GMT):
re: my first question from last night. i'm guessing that having the peer service named `peer0-org1` works because CORE_PEER_TLS_ENABLED is false. is that right?
foleymic (Mon, 12 Feb 2018 19:38:17 GMT):
Has joined the channel.
Senthil1 (Tue, 13 Feb 2018 11:18:12 GMT):
Has joined the channel.
JayJong (Wed, 14 Feb 2018 03:59:37 GMT):
Hi guys, I have set up fabric using kubernetes and the fabric SDK for node.js to talk to the blockchain. I am testing the performance of fabric by timing the start of invoking a single transaction on node A to the end of quering the same transaction on node B. I wrote a shell script to run the invoke.js and query.js in the fabcar sample. The result was 3.9 seconds for the writing and reading to complete.
I have 2 questions:
1. Am i testing it correctly? Is that the right way to calculate tx per second? Fabric is said to be able to do 3500 tx per second and ethereum is able to do 15 tx per second. Should I be pushing 10,000 or more transactions to the blockchain and see how long it takes and divide 10,000 by time taken?
2. Why does it take 3.9 seconds to invoke and query using the node.js sdk? Is it because of the javascript code or the authentication takes sometime?
mamtabhardwaj12 (Wed, 14 Feb 2018 12:32:12 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=AJnvJFs2KHfrn9trX) @Luke_Chen I am not getting kube-dns-xxxxx
RobertDiebels (Thu, 15 Feb 2018 09:42:44 GMT):
Hey guys, I released my project on github yesterday. I had some issues with moving the NPM package for the CLI to an account dedicated to the project so the NPM package is not install-able yet.
RobertDiebels (Thu, 15 Feb 2018 09:43:19 GMT):
In the meanwhile you can clone the code and just run the commands with `node .
RobertDiebels (Thu, 15 Feb 2018 09:43:19 GMT):
In the meanwhile you can clone the code and just run the commands with `node .
RobertDiebels (Thu, 15 Feb 2018 09:43:25 GMT):
Feel free to check it out https://github.com/kubechain/kubechain
RobertDiebels (Thu, 15 Feb 2018 09:44:52 GMT):
Especially Linux/OSX users since I've tested mainly on Windows. However I wrote the code to run on multiple platforms so I'm fairly certain it will run.
RobertDiebels (Thu, 15 Feb 2018 09:44:57 GMT):
Either way let me know :D
KillerGasy (Thu, 15 Feb 2018 12:08:25 GMT):
Has joined the channel.
tsnyder (Thu, 15 Feb 2018 12:36:18 GMT):
@ublubu - If you look at the command file - setup-dev/fabric-gen/channel-join.sh you will see the commands I am using to join the channels. These are run from within the CLI container as described within the doc.
tsnyder (Thu, 15 Feb 2018 12:39:08 GMT):
@ublubu - yes TLS is turned off. All of the network DNS is done within the kubernetes environment for the Peers, Orderers, CouchDB, sdk, etc., via the services.
RobertDiebels (Thu, 15 Feb 2018 16:52:29 GMT):
Just published the CLI on NPM. Can be found here: https://www.npmjs.com/package/kubechain
vanitas92 (Thu, 15 Feb 2018 17:32:58 GMT):
Hi guys, I would like to know how to handle the restart of a peer pod so that it recovers automatically after a crash in kubernetes. By recovering i mean, join the channels it was previously joined before the crash, recover the installation of chaincodes previously installed and instantiate or invoke the chaincodes installed. I think one way is using persistent volume at path `/var/hyperledger/` of containers. Is this the best way or are there others that are better? Let me know! Thank you all guys, you rock!
ublubu (Thu, 15 Feb 2018 18:26:02 GMT):
thanks, @tsnyder. i got it working yesterday. the key for me was to make sure TLS was turned off everywhere and to configure Docker to use kube-dns.
LuckyMacky (Fri, 16 Feb 2018 09:14:54 GMT):
Has joined the channel.
paul.sitoh (Fri, 16 Feb 2018 12:06:19 GMT):
Has joined the channel.
dv29 (Sat, 17 Feb 2018 08:39:09 GMT):
Has joined the channel.
mrkiouak (Sat, 17 Feb 2018 14:29:55 GMT):
Has joined the channel.
leezie (Sat, 17 Feb 2018 15:24:00 GMT):
Has joined the channel.
nicolapaoli (Mon, 19 Feb 2018 14:15:37 GMT):
Has joined the channel.
DannyWong (Wed, 21 Feb 2018 09:34:01 GMT):
in K8S deployment, I understand that we need to have different set of deployment YAML for each peer due to different kinds of config (kinda Stateful Set).
How about Orderer (not Kafka) Service node, can we have use replica > 1 in K8S Deployment???
Luke_Chen (Wed, 21 Feb 2018 16:30:15 GMT):
@DannyWong It sure can use more than 1 replica orderer.
Luke_Chen (Wed, 21 Feb 2018 16:30:15 GMT):
@DannyWong It's sure that we can use more than 1 replica orderer.
Luke_Chen (Wed, 21 Feb 2018 16:30:15 GMT):
@DannyWong It's sure that we can use more than 1 replica of orderer.
DannyWong (Thu, 22 Feb 2018 01:26:08 GMT):
@Luke_Chen ok let me try also. Thanks for your help
tupt (Thu, 22 Feb 2018 03:22:11 GMT):
there are many problems with deploying hyperledger on kubernetes, including docker images sharing, host resolving... for easy installation stuff, I think you can use expect or sshpass to automate the synchronization between nodes like save and load image, and share host file(map ip to hostname) then start like normal setup.
Taffies (Thu, 22 Feb 2018 06:23:43 GMT):
Hello! Anyone here successfully deployed a fabric network using orderer type kafka on kubernetes? :)
yoko (Sun, 25 Feb 2018 02:44:35 GMT):
Has joined the channel.
yzhivkov (Sun, 25 Feb 2018 18:47:44 GMT):
Has joined the channel.
chandrakanthm (Wed, 28 Feb 2018 09:56:48 GMT):
Has joined the channel.
dampuero (Wed, 28 Feb 2018 15:58:00 GMT):
Has joined the channel.
iperrota (Wed, 28 Feb 2018 21:36:59 GMT):
Has joined the channel.
iperrota (Wed, 28 Feb 2018 21:40:31 GMT):
Hi! I have a problem to exec create_channel.sh, the spript didn't finish and print the message Wating for Createchanel to be completed
joaquimpedrooliveira (Thu, 01 Mar 2018 13:47:44 GMT):
@Taffies we did it in our company
joaquimpedrooliveira (Thu, 01 Mar 2018 13:47:56 GMT):
(deployed a fabric network using orderer type kafka on kubernetes)
alexvicegrab (Fri, 02 Mar 2018 15:26:05 GMT):
Has joined the channel.
bmalavan (Fri, 02 Mar 2018 15:54:46 GMT):
Has joined the channel.
igetgames (Fri, 02 Mar 2018 17:42:44 GMT):
Has joined the channel.
joaquimpedrooliveira (Fri, 02 Mar 2018 20:05:03 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=5pacZyweM8qbxFWSE) @Taffies What problems are you facing?
ashishapy (Sat, 03 Mar 2018 19:54:20 GMT):
Has joined the channel.
magicliang (Tue, 06 Mar 2018 10:24:25 GMT):
Has joined the channel.
mastersingh24 (Tue, 06 Mar 2018 10:33:14 GMT):
Not quite sure I agree with this comment? Can you provide more details about issues you are seeing?
Of course chaincode deployment *can* be an issue.
And if you are trying to run an entire network (multiple orgs, etc) / sandbox things might be a little tricky (although this is just standard stuff to make distributed things communicate).
If you are simply trying to deploy a peer or an ordering service, not sure there are many issues
(https://chat.hyperledger.org/channel/fabric-kubernetes?msg=sbvEM9A6Sn9en72Tr) @tupt
dave.enyeart (Tue, 06 Mar 2018 17:06:03 GMT):
Has joined the channel.
rjones (Tue, 06 Mar 2018 18:13:44 GMT):
mastersingh24
rjones (Tue, 06 Mar 2018 18:13:51 GMT):
dave.enyeart
Glen (Thu, 08 Mar 2018 02:29:12 GMT):
Hi, anybody tried kafka deployment on kuberntes?
gbolarhan (Thu, 08 Mar 2018 16:47:29 GMT):
Has joined the channel.
DavidWalter (Sat, 10 Mar 2018 20:57:53 GMT):
Has joined the channel.
tkueda (Tue, 13 Mar 2018 01:19:52 GMT):
Has joined the channel.
divudivyansh (Tue, 13 Mar 2018 04:33:44 GMT):
Has joined the channel.
Vthot4 (Tue, 13 Mar 2018 13:40:45 GMT):
Has joined the channel.
yopep (Wed, 14 Mar 2018 09:08:09 GMT):
Has joined the channel.
davidkhala (Thu, 15 Mar 2018 04:13:05 GMT):
Has joined the channel.
davidkhala (Thu, 15 Mar 2018 04:16:06 GMT):
@mastersingh24 Dear Gari, as a new participant here, I want to ask is there a preference between k8s and docker-swarm when automate fabric deploying on cluster?
mhagelstrom (Thu, 15 Mar 2018 13:47:44 GMT):
Has joined the channel.
mhagelstrom (Thu, 15 Mar 2018 13:48:53 GMT):
Hello guys. Is there any documentation or guide I can use to size each peer node of the network?
Huijian (Fri, 16 Mar 2018 09:03:12 GMT):
Has joined the channel.
govinda-attal (Fri, 16 Mar 2018 13:22:18 GMT):
Has joined the channel.
govinda-attal (Fri, 16 Mar 2018 13:23:17 GMT):
Hi All, I have been playing around with Hyperledger to make it run on Kubernetes. And I was successful to do so. The only thing which I was not happy with the solution for the container that was spun up when chaincode is instantiated by the peer. Kubernetes is simply not aware of this container as it was not started by Kubernetes and by the peer. And to make the peer and chaincode talk to each other I had to update the docker daemon running on the kubernetes node with dns server ip address of the kube-dns service.
govinda-attal (Fri, 16 Mar 2018 13:23:35 GMT):
Is it possible to instantiate a chaincode in a way where kubernetes is aware of the container of the chaincode. And also chaincode container is able to talk to peer in a seamless fashion rather than updating docker daemon process of the node within kubernetes cluster
rajeshlc (Fri, 16 Mar 2018 13:26:58 GMT):
Has joined the channel.
govinda-attal (Fri, 16 Mar 2018 14:35:42 GMT):
@tupt I agree with @mastersingh24 . Sharing small POC on ... https://github.com/govinda-attal/hl-on-k8s ... But I don't like that work-around and I am seeking some advice to the community in general
govinda-attal (Fri, 16 Mar 2018 14:35:42 GMT):
@tupt I agree with @mastersingh24 . Sharing small POC on ... https://github.com/govinda-attal/hl-on-k8s ... But I don't like that work-around and I am seeking some advice from the community
sillysachin (Fri, 16 Mar 2018 17:58:55 GMT):
Has joined the channel.
cbf (Sat, 17 Mar 2018 18:49:00 GMT):
@davidkhala check out project Cello - right now, the focus is on K8s
cbf (Sat, 17 Mar 2018 18:49:12 GMT):
see the #cello channel
ashishapy (Sun, 18 Mar 2018 08:25:52 GMT):
@govinda-attal see good discussion here https://jira.hyperledger.org/browse/FAB-3339 about K8s & Chaincode
ascatox (Sun, 18 Mar 2018 15:48:50 GMT):
Has joined the channel.
Gerard9494 (Sun, 18 Mar 2018 20:21:54 GMT):
Has joined the channel.
davidkhala (Mon, 19 Mar 2018 01:05:16 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=ptsCoFnNvzKsnhq3d) @cbf Thanks, and actually I have tried cello and not satisfied with it. still seeking for alternative idea.
yopep (Mon, 19 Mar 2018 03:18:22 GMT):
In a kubernetes environment, should cello-operator-dashboard be part of the cluster? Or should it be deployed outside of the kubernetes cluster?
govinda-attal (Mon, 19 Mar 2018 05:42:12 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=rx4iTekMen6EX2Zrd) @ashishapy Thank you very much
alexvicegrab (Mon, 19 Mar 2018 17:25:07 GMT):
I'm setting up HLF on Kubernetes (using Azure AKS), but get stuck at the point of creating a channel from a host:
peer channel create -o 10.0.69.21:31010 -c composerchannel -f /shared/composer-channel.tx
```
2018-03-19 16:50:15.657 UTC [msp] GetLocalMSP -> DEBU 001 Returning existing local MSP
2018-03-19 16:50:15.658 UTC [msp] GetDefaultSigningIdentity -> DEBU 002 Obtaining default signing identity
Error: failed to create deliver client: orderer client failed to connect to 10.0.69.21:31010: failed to create new connection: context deadline exceeded
```
Installing ping on the host, and trying to ping related services, I'm only able to ping the IP/DNS of the Peer container, but not that of the Orderer, etc., even though I can easily ssh into each of these from outside.
Has anyone faced similar pod to pod connectivity issues, and how have you resolved them?
Thanks!
yogeshquick (Tue, 20 Mar 2018 07:44:40 GMT):
Has joined the channel.
Varun2887 (Tue, 20 Mar 2018 07:50:52 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=KwarRHsWmYTL7ozqu) @tongli Do we have anything ready on AWS?
Varun2887 (Tue, 20 Mar 2018 07:50:58 GMT):
for kubernet+composer setup
RobertDiebels (Tue, 20 Mar 2018 10:38:50 GMT):
@Varun2887 I'm working on getting a kubernetes cluster working on AWS right now.
RobertDiebels (Tue, 20 Mar 2018 10:41:02 GMT):
I've build a tool called Kubechain which transforms blockchain congfiguration to kubernetes configuration. It's pre-alpha right now but you can check it out either way: https://www.npmjs.com/package/kubechain
RobertDiebels (Tue, 20 Mar 2018 10:41:31 GMT):
It's build using NodeJS so it should work regardless of which platform you're working on.
RobertDiebels (Tue, 20 Mar 2018 10:42:11 GMT):
Source can be found here: https://github.com/kubechain/kubechain
RobertDiebels (Tue, 20 Mar 2018 10:43:36 GMT):
The main reason I build the tool is because all other tools I could find were highly complex for such a simple thing and didn't work on Windows.
RobertDiebels (Tue, 20 Mar 2018 10:46:15 GMT):
For example I can't use Cello because it requires `make` to build a Master node. Which is not available under windows unless I have a VS Studio license or I install something to mimic that functionality.
RobertDiebels (Tue, 20 Mar 2018 10:49:45 GMT):
Both of which are undesirable. Personally I feel that using a Makefile to build a supposedly cross-platform application is a big mistake. Since it provides a barrier to people who want to test/build on platforms other than Linux.
RobertDiebels (Tue, 20 Mar 2018 10:52:30 GMT):
Fabric also uses a Makefile and so does Burrow. I don't think it would be that much effort to use a go file to build the source especially since go is cross-platform and Makefiles are not.
RobertDiebels (Tue, 20 Mar 2018 10:52:30 GMT):
Fabric uses a Makefile and so does Burrow. I don't think it would be that much effort to use a go file to build the source especially since go is cross-platform and Makefiles are not.
Varun2887 (Tue, 20 Mar 2018 11:08:42 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=oCfoEfe6GmzdEDSre) @RobertDiebels, Admins is there anything fully functional available for this?
alexvicegrab (Tue, 20 Mar 2018 11:12:03 GMT):
Solved my own problem, I was using NodePort services (since I was using the ibm-composer-network as inspiration) and switched to ClusterIP service, which made it work on Azure AKS.
alexvicegrab (Tue, 20 Mar 2018 11:12:27 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=b6pxuuaNFY4cC9gYF)
Solved my own problem, I was using NodePort services (since I was using the ibm-composer-network as inspiration) and switched to ClusterIP service, which made it work on Azure AKS.
alexvicegrab (Tue, 20 Mar 2018 11:12:27 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=b6pxuuaNFY4cC9gYF)
Solved my own problem, I was using NodePort services (since I was using the ibm-composer-network as inspiration) and switched to ClusterIP service, which made it work on Azure AKS.
ShikarSharma (Tue, 20 Mar 2018 22:43:27 GMT):
Has joined the channel.
bh4rtp (Wed, 21 Mar 2018 17:26:21 GMT):
Has joined the channel.
bh4rtp (Wed, 21 Mar 2018 17:26:54 GMT):
hi, does fabric support kubernets now?
RobertDiebels (Wed, 21 Mar 2018 19:36:42 GMT):
@Varun2887 I think that 1.1.0-alpha fixes a few issues important for interop with Kubernetes. Among which reading sym-links
sillysachin (Thu, 22 Mar 2018 13:24:54 GMT):
in Release is out 1.1 the examples directory with e2e_cli has a lot of dependencies on fabric build , can it run independently
StevenSiahetiong (Fri, 23 Mar 2018 02:10:01 GMT):
Has joined the channel.
mwagner (Fri, 23 Mar 2018 13:42:22 GMT):
does anyone have Fabric running on OpenShift ?
BryanSparks (Fri, 23 Mar 2018 14:00:48 GMT):
Has joined the channel.
remyabdullahi (Fri, 23 Mar 2018 19:50:31 GMT):
Has joined the channel.
chenjun-bj (Mon, 26 Mar 2018 01:53:39 GMT):
Has joined the channel.
AshishMishra 1 (Tue, 27 Mar 2018 09:40:42 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=isihTegi8y5JwT5kC) @mwagner Interested to know if anyone succeeded.
zian.yusuf (Tue, 27 Mar 2018 09:45:20 GMT):
Has joined the channel.
spruce (Tue, 27 Mar 2018 17:36:22 GMT):
Has joined the channel.
MonnyClara (Wed, 28 Mar 2018 12:56:54 GMT):
Has joined the channel.
sukritVisa (Wed, 28 Mar 2018 21:06:29 GMT):
Has joined the channel.
pmcosta1 (Thu, 29 Mar 2018 10:45:46 GMT):
Has left the channel.
MonnyClara (Thu, 29 Mar 2018 13:28:08 GMT):
Hello team!
I’m currently deploying an HLF blockchain on a K8s cluster. But I have some trouble to instantiate a chaincode : when I run my ` peer chaincode instantiate` command I have the following error `Error: Error endorsing chaincode: rpc error: code = Unknown desc = Timeout expired while starting chaincode `
Did someone experienced this before ?
Luke_Chen (Fri, 30 Mar 2018 14:44:16 GMT):
@MonnyClara It may caused by the dns resolve error
MonnyClara (Fri, 30 Mar 2018 14:45:52 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=ER9ne24QkKP37awRv) @Luke_Chen Even if they can ping each other ? I mean that peer1.org1 can ping peer1.org2 ?
So I supposed that the dns resolution was okey ... I'm going to re-check !
Luke_Chen (Fri, 30 Mar 2018 14:48:13 GMT):
When you instantiate a chaincode, the chaincode container will be created by docker daemon, which is out of kubernetes
MonnyClara (Fri, 30 Mar 2018 15:05:14 GMT):
Hoooo okey, I got it ... Have you find a way to solve this problem ?
Luke_Chen (Fri, 30 Mar 2018 15:12:29 GMT):
refer this link https://labs.vmware.com/flings/blockchain-on-kubernetes
Luke_Chen (Fri, 30 Mar 2018 15:12:55 GMT):
download installation guide
vvnick (Sat, 31 Mar 2018 11:31:34 GMT):
Has left the channel.
anzalbeg (Sun, 01 Apr 2018 14:21:13 GMT):
Has joined the channel.
richzhao (Sun, 01 Apr 2018 15:18:14 GMT):
Has joined the channel.
yazanalhjaj (Mon, 02 Apr 2018 07:51:15 GMT):
Has joined the channel.
joaquimpedrooliveira (Mon, 02 Apr 2018 13:58:55 GMT):
Hello, everybody. We're running Fabric over Kubernetes and an update in Kubernetes apparently impacted in chaincode instantiation. After upgrading to kubernetes `1.7.15` chaincode instantation started to fail with message:
```Failed to generate platform-specific docker build: Error returned from build: 1 "can't load package: package chaincode/chaincode_example02: cannot find package
```
joaquimpedrooliveira (Mon, 02 Apr 2018 13:59:55 GMT):
The chaincode file exists and is mounted as a `config-map` . We noticed that this k8s version introduced a change that mounts `config-maps` as a **read-only** filesystem, that seems to be error cause.
joaquimpedrooliveira (Mon, 02 Apr 2018 14:00:38 GMT):
When we added the chaincode to be part of docker image, and stopped using the configmap, it worked.
joaquimpedrooliveira (Mon, 02 Apr 2018 14:01:39 GMT):
I cannot see why the "cli" pod needs write access to the chaincode dir
joaquimpedrooliveira (Mon, 02 Apr 2018 14:02:15 GMT):
Another test that worked was to log into the pod and move the chaincode file to a new dir, created in another volume with read/write permissions
joaquimpedrooliveira (Mon, 02 Apr 2018 14:02:53 GMT):
Any tips why chaincode instantiation needs to have write acccess to the "CLI"/fabric-tools filesystem?
coffeeplease (Mon, 02 Apr 2018 20:58:01 GMT):
Has joined the channel.
Othman.Darwish (Wed, 04 Apr 2018 07:29:58 GMT):
Has joined the channel.
MonnyClara (Wed, 04 Apr 2018 09:11:13 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=N9xkJxReHKgkEJRXm) @Luke_Chen Thank you for the tip ! The guide was just perfect for the error I had
daygee (Thu, 05 Apr 2018 14:23:11 GMT):
hi everyone
daygee (Thu, 05 Apr 2018 14:23:54 GMT):
so I'm trying dind to resolve the issue with instantiation
daygee (Thu, 05 Apr 2018 14:24:54 GMT):
but because I'm using a company kubernetes cluster, there are certain restrictions, such as images cannot be pulled from anywhere but the company repo
daygee (Thu, 05 Apr 2018 14:35:23 GMT):
so whenever the instantiation command is called, the peer tries to pull form docker repo directly
but it can't because connection will be refused
I want to know if there was a way to configure the peer to change where it pulls the ccenv image from when instantiating
daygee (Thu, 05 Apr 2018 14:35:23 GMT):
so whenever the instantiation command is called, the peer tries to pull from docker hub directly
but it can't because connection will be refused
I want to know if there was a way to configure the peer to change where it pulls the ccenv image from when instantiating
saikiran458 (Thu, 05 Apr 2018 18:41:42 GMT):
Has joined the channel.
alexvicegrab (Fri, 06 Apr 2018 15:13:05 GMT):
daygee, you can update the deployment.yaml:
```
containers:
- name: peer
image: YOUR_REPOSITORY/YOUR_IMAGE
```
alexvicegrab (Fri, 06 Apr 2018 15:13:05 GMT):
@daygee, you can update the deployment.yaml:
```
containers:
- name: peer
image: YOUR_REPOSITORY/YOUR_IMAGE
```
kkado (Sat, 07 Apr 2018 04:12:43 GMT):
Has joined the channel.
bh4rtp (Sat, 07 Apr 2018 09:18:58 GMT):
is there a demo to build an example fabric network using kubernetes?
daygee (Mon, 09 Apr 2018 14:57:30 GMT):
@bh4rtp there is no demo, but there are walkthroughs
dokany (Mon, 09 Apr 2018 22:07:50 GMT):
Has joined the channel.
eroldan (Tue, 10 Apr 2018 05:10:20 GMT):
Has joined the channel.
Luke_Chen (Tue, 10 Apr 2018 07:31:28 GMT):
@daygee can you pull from dockerhub and then upload to your company repo?
Luke_Chen (Tue, 10 Apr 2018 07:32:01 GMT):
these is no way to config where to pulls the ccenv image
pankajcheema (Tue, 10 Apr 2018 08:41:47 GMT):
Has joined the channel.
pankajcheema (Tue, 10 Apr 2018 08:49:16 GMT):
Hi experts
pankajcheema (Tue, 10 Apr 2018 08:49:37 GMT):
Anyone tried setting up a physical network using docker swarms?
greg.haskins (Wed, 11 Apr 2018 02:14:41 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=QohsyHjMEFbxcWz6F) @bh4rtp @daygee https://gerrit.hyperledger.org/r/#/c/12159/
vikramjit (Wed, 11 Apr 2018 10:56:40 GMT):
Has joined the channel.
alexvicegrab (Wed, 11 Apr 2018 19:33:45 GMT):
@bh4rtp there are some Helm charts here, but they are not very up to date and need customisation: https://github.com/IBM-Blockchain/ibm-container-service
alexvicegrab (Wed, 11 Apr 2018 19:33:45 GMT):
@bh4rtp there are some K8S/Helm charts here, but they are not very up to date and need customisation: https://github.com/IBM-Blockchain/ibm-container-service
bh4rtp (Thu, 12 Apr 2018 00:31:32 GMT):
@greg.haskins @alexvicegrab appreciate your help!
bh4rtp (Thu, 12 Apr 2018 02:43:40 GMT):
@greg.haskins i have downloaded the cluster. `make kubernetes` succeeded. but run `kubectl create -f build/kubernetes.yaml --save-config` failed.
```error: error validating "build/kubernetes.yaml": error validating data: invalid object to validate; if you choose to ignore these errors, turn validation off with --validate=false```
bh4rtp (Thu, 12 Apr 2018 02:43:40 GMT):
@greg.haskins i have downloaded the cluster example with kubernetes. `make kubernetes` succeeded. but run `kubectl create -f build/kubernetes.yaml --save-config` failed.
```error: error validating "build/kubernetes.yaml": error validating data: invalid object to validate; if you choose to ignore these errors, turn validation off with --validate=false```
bh4rtp (Thu, 12 Apr 2018 02:51:54 GMT):
above error was cleared by executing `dos2unix` for every text source files.
greg.haskins (Thu, 12 Apr 2018 02:52:44 GMT):
@bh4rtp can you post output from "kubectl --version"
bh4rtp (Thu, 12 Apr 2018 02:53:38 GMT):
```Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"", Minor:"", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2018-01-26T19:04:38Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"}
```
bh4rtp (Thu, 12 Apr 2018 02:56:18 GMT):
Clipboard - 2018年4月12日上午10点55分
bh4rtp (Thu, 12 Apr 2018 02:57:37 GMT):
@greg.haskins all the deployments failed for secrets not found.
bh4rtp (Thu, 12 Apr 2018 02:59:42 GMT):
Clipboard - 2018年4月12日上午10点59分
bh4rtp (Thu, 12 Apr 2018 08:16:04 GMT):
@greg.haskins what `kubectl` version are you using?
bh4rtp (Thu, 12 Apr 2018 09:51:32 GMT):
i concatenate all secrets into one single line and it does work.
greg.haskins (Fri, 13 Apr 2018 00:17:40 GMT):
I might have a fix that I need to push
moucx (Fri, 13 Apr 2018 11:45:42 GMT):
Has joined the channel.
bh4rtp (Fri, 13 Apr 2018 15:31:31 GMT):
@greg.haskins i modified kubernetes/generate-secrets.sh with `data:
config.tgz: $(cat $FILE | base64 | tr -d '\r\n')```
bh4rtp (Fri, 13 Apr 2018 15:35:36 GMT):
@greg.haskins i modified `kubernetes/generate-secrets.sh` with
```data:
config.tgz: $(cat $FILE | base64 | tr -d '\r\n')```
so does `kubernetes/generate-client-config.sh`
and then `kubectl create -f build/kubernetes.yaml --save-config` runs ok. but all the peer nodes crashed with error
```Readiness probe failed: Get http://172.17.0.9:5984/: dial tcp 172.17.0.9:5984: getsockopt: connection refused
Back-off restarting failed container```
bh4rtp (Fri, 13 Apr 2018 15:38:40 GMT):
Clipboard - 2018年4月13日晚上11点38分
alexvicegrab (Fri, 13 Apr 2018 18:46:17 GMT):
What does your readiness probe look like?
Taffies (Mon, 16 Apr 2018 02:33:36 GMT):
hi! this is weird but may i ask if anyone keeps having to re-join channel/install/instantiate chaincode for each org? i did it two days ago but today it showed me that my all my orgMSP is unknown and i have to repeat the steps that i did
Taffies (Mon, 16 Apr 2018 02:33:36 GMT):
hi! this is weird but may i ask if anyone keeps having to re-join channel/install chaincode for each org? i did it two days ago but today it showed me that my all my orgMSP is unknown and i have to repeat the steps that i did
neharprodduturi (Mon, 16 Apr 2018 06:55:15 GMT):
Has joined the channel.
MonnyClara (Mon, 16 Apr 2018 07:38:31 GMT):
@Taffies Yeah ! I have the same issue, everytime a peer container is crashing, you have to re-join and re-install .. A way to fix this ?
nickgaski (Mon, 16 Apr 2018 17:09:05 GMT):
Has joined the channel.
Taffies (Tue, 17 Apr 2018 03:13:47 GMT):
@monnyclara mine doesn't even crash. the next morning i wake up and log in to my VM and the pod is still running perfectly fine, but i just need to rejoin and reinstall again. i can't figure out why
ascatox (Tue, 17 Apr 2018 09:55:38 GMT):
Hi All!!! Has someone a good tutorial to start with K8s and Fabric ?
MonnyClara (Tue, 17 Apr 2018 10:00:46 GMT):
@ascatox Check this one : https://hackernoon.com/how-to-deploy-hyperledger-fabric-on-kubernetes-1-a2ceb3ada078
ascatox (Tue, 17 Apr 2018 10:01:00 GMT):
:thumbsup:
RobertDiebels (Tue, 17 Apr 2018 10:20:21 GMT):
@ascatox I've made a NodeJS CLI that installs Fabric on either Minikube or GCE.
RobertDiebels (Tue, 17 Apr 2018 10:20:46 GMT):
Check it out here: https://www.npmjs.com/package/kubechain
RobertDiebels (Tue, 17 Apr 2018 10:22:05 GMT):
I'm currently working on expanding the feature set.
RobertDiebels (Tue, 17 Apr 2018 10:22:26 GMT):
Right now it only creates a very basic Fabric cluster but that should suffice for testing.
ascatox (Tue, 17 Apr 2018 15:05:27 GMT):
This sounds really interesting!!! I'll try it
RobertDiebels (Tue, 17 Apr 2018 16:11:29 GMT):
Please do. Don't be afraid to post a bug-report if you find anything. I might not be able to solve it that fast but I'll try :D :thumbsup:
Chandoo (Thu, 19 Apr 2018 17:52:20 GMT):
Has joined the channel.
remmeier (Fri, 20 Apr 2018 06:35:38 GMT):
Has joined the channel.
Taffies (Fri, 20 Apr 2018 09:17:25 GMT):
does anyone else face the problem where your peer keeps having to rejoin channel & installing chaincode on them even though your pods haven't crashed at all? or any idea why that might be happening?
Taffies (Fri, 20 Apr 2018 09:17:39 GMT):
it's frustrating :<
Taffies (Fri, 20 Apr 2018 09:26:10 GMT):
i'm using kafka as my orderer if it is of significance
raccoonrat (Sun, 22 Apr 2018 06:25:48 GMT):
Has joined the channel.
toesterdahl (Sun, 22 Apr 2018 20:39:08 GMT):
Has joined the channel.
rcheuk (Tue, 24 Apr 2018 20:04:50 GMT):
Has joined the channel.
kipharris (Wed, 25 Apr 2018 13:32:21 GMT):
DNS
Rednish (Wed, 25 Apr 2018 16:12:13 GMT):
Has joined the channel.
Taffies (Thu, 26 Apr 2018 02:47:18 GMT):
anyone successfully upgraded their images to 1.1.0 on kubernetes? i'm trying to do so but my peer pods always crash whenever i change my image to 1.1.0 for the peers
Taffies (Thu, 26 Apr 2018 02:53:21 GMT):
hi! anyone successfully upgraded their images to 1.1.0? i'm trying to upgrade but my peer pods keep crashing
Taffies (Thu, 26 Apr 2018 02:53:35 GMT):
Screen Shot 2018-04-26 at 10.52.52 AM.png
Taffies (Thu, 26 Apr 2018 02:53:49 GMT):
Screen Shot 2018-04-26 at 10.43.48 AM.png
Taffies (Thu, 26 Apr 2018 05:13:07 GMT):
hi for those who got the same error as me above: i added an additional parameter in the peer config file under environment:
- name: CORE_PEER_CHAINCODELISTENADDRESS
value: "0.0.0.0:7052"
and i was able to solve the error and get my pods working. :grinning:
anthonyk (Thu, 26 Apr 2018 05:57:57 GMT):
Has joined the channel.
maestrus (Thu, 26 Apr 2018 08:29:52 GMT):
Has joined the channel.
chainsaw (Fri, 27 Apr 2018 15:52:11 GMT):
Has joined the channel.
acbellini (Tue, 01 May 2018 21:10:05 GMT):
Has joined the channel.
IgorSim (Wed, 02 May 2018 09:49:26 GMT):
Has joined the channel.
carlcraig (Wed, 02 May 2018 20:56:02 GMT):
Has joined the channel.
kevin-s-wang (Thu, 03 May 2018 02:36:06 GMT):
Has joined the channel.
shabscan (Fri, 04 May 2018 16:48:57 GMT):
Has joined the channel.
shabscan (Fri, 04 May 2018 16:52:16 GMT):
hi, any idea how to expose the public ip for the composer-rest-server with kubernetes on IBM cloud ? Was able to get to playground instance , but Rest server is proving hard :(
Starseven (Tue, 08 May 2018 11:56:08 GMT):
Has joined the channel.
jeremyphelps (Tue, 08 May 2018 15:20:35 GMT):
Has joined the channel.
tcskill (Tue, 08 May 2018 19:03:21 GMT):
Has joined the channel.
adave (Wed, 09 May 2018 09:08:10 GMT):
Has joined the channel.
MrMoneyChanger (Wed, 09 May 2018 21:44:06 GMT):
Has joined the channel.
titoe218 (Fri, 11 May 2018 02:01:13 GMT):
Has joined the channel.
versus (Mon, 14 May 2018 09:05:55 GMT):
Has joined the channel.
Switch2Logic (Wed, 16 May 2018 07:33:10 GMT):
Has joined the channel.
BhaveshPatadiya (Fri, 18 May 2018 03:04:01 GMT):
Has joined the channel.
BhaveshPatadiya (Fri, 18 May 2018 03:07:32 GMT):
Hi, I am new to kubernetes and would like to integrate the same in fabric. What would be the good starting point fro me? Any help and suggestions would be appreciated.
BhaveshPatadiya (Fri, 18 May 2018 03:47:10 GMT):
I also want to ask whether there is any starting documentation from where I can start using Kubernetes for fabric app I'm developing?
MonnyClara (Fri, 18 May 2018 07:19:24 GMT):
A good start would be to learn to use Kubernetest ;)
Then you can check for this : https://hackernoon.com/how-to-deploy-hyperledger-fabric-on-kubernetes-1-a2ceb3ada078
MonnyClara (Fri, 18 May 2018 07:19:24 GMT):
A good start would be to learn to use Kubernetes ;)
Then you can check for this : https://hackernoon.com/how-to-deploy-hyperledger-fabric-on-kubernetes-1-a2ceb3ada078
ascatox (Fri, 18 May 2018 14:50:35 GMT):
Hi I'm investigating the usage of K8s for my fabric installation.
ascatox (Fri, 18 May 2018 14:51:02 GMT):
I'd like to create a pod containing all peers, in order to create a load balancing
ascatox (Fri, 18 May 2018 14:51:02 GMT):
I'd like to create a pod containing all peers, in order to create a load balancing of the requests
ascatox (Fri, 18 May 2018 14:51:02 GMT):
I'd like to create a pod containing all peers, in order to create a load balancing of the requests.
MonnyClara (Fri, 18 May 2018 15:12:46 GMT):
@ascatox With this kind a setup, if your pod is crashing all your peer will crash too right ? If so, we got a single point of failure I think ..
ascatox (Fri, 18 May 2018 15:13:59 GMT):
ok but if I have 1000 clients making transactions to the blockchain how I can improve the load on my peers ?
ascatox (Fri, 18 May 2018 15:15:31 GMT):
if I'm not wrong at the moment the address of the peer is static, every client should connect to a peer and knows its address
ascatox (Fri, 18 May 2018 15:15:31 GMT):
Which version ???
alexvicegrab (Fri, 18 May 2018 23:36:50 GMT):
You should use a service to loadbalance to the set of peers, but this will only work if your endorsement policy requires a single peer.
alexvicegrab (Fri, 18 May 2018 23:36:50 GMT):
You should use a service to loadbalancer to the set of peers, but this will only work if your endorsement policy requires a single peer.
alexvicegrab (Fri, 18 May 2018 23:36:50 GMT):
You could use a service to loadbalancer to the set of peers, but this will only work if your endorsement policy requires a single peer.
yacovm (Sat, 19 May 2018 21:57:40 GMT):
@ascatox - starting from the next version, the application won't need to know any address of the endorsing peers at all
ascatox (Mon, 21 May 2018 06:19:21 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=nMKDnjNeL9eyYavNx) @yacovm Which version are you speaking about ?
yacovm (Mon, 21 May 2018 08:01:47 GMT):
next fabric version of course @ascatox
ascatox (Mon, 21 May 2018 08:42:03 GMT):
1.2?
JayJong (Mon, 21 May 2018 09:33:29 GMT):
hi guys, im trying to setup fabric on kubernetes but im unsure if the kafka and zookeeper brokers are setup in a VM(virtual machine) or on separate VMs? any help is appreciated!
vanitas92 (Mon, 21 May 2018 19:27:56 GMT):
Hi guys do you know when would be available the instantiation of chaincodes fully integrated within kubernetes? I read that its planned to be available at 1.2?
MonnyClara (Tue, 22 May 2018 06:39:18 GMT):
@JayJong It can be on the same VM or in separate, as long as they can communicate
sukrit.handa@gmail.com (Wed, 23 May 2018 03:01:04 GMT):
Has joined the channel.
simonghrt (Wed, 23 May 2018 12:21:44 GMT):
Has joined the channel.
sukrit.handa@gmail.com (Wed, 23 May 2018 23:06:07 GMT):
running into issues with kafka deployment on k8s. the brokers and zookeeper nodes are communicating fine but the topic partition is unable to find a leader getting the following for the orderer log:
[sarama] 2018/05/23 22:57:46.349687 client.go:629: client/metadata fetching metadata for [testchainid] from broker kafka1.0018875-dev-business:9092
sukrit.handa@gmail.com (Wed, 23 May 2018 23:06:07 GMT):
running into issues with kafka deployment on k8s. the brokers and zookeeper nodes are communicating fine but the topic partition is unable to find a leader getting the following for the orderer log:
[sarama] 2018/05/23 22:57:46.349687 client.go:629: client/metadata fetching metadata for [testchainid] from broker kafka1.0018875-dev-business:9092[sarama] 2018/05/23 22:57:46.352899 client.go:640: client/metadata found some partitions to be leaderless
MonnyClara (Thu, 24 May 2018 07:11:46 GMT):
@sukrit.handa@gmail.com I'm not an expert, but you may have issue with the leader election
I think I used a env variable like `KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE=false`
vikramjit (Thu, 24 May 2018 10:49:21 GMT):
Has left the channel.
sukrit.handa@gmail.com (Thu, 24 May 2018 16:06:43 GMT):
yea i have that :/
sukrit.handa@gmail.com (Thu, 24 May 2018 16:06:53 GMT):
anyways will continue to debug
sukrit.handa@gmail.com (Thu, 24 May 2018 20:54:44 GMT):
got it to work kafka brokers on k8s needs to have the following env
- name: KAFKA_ADVERTISED_HOST_NAME
value: "kafka1"
- name: KAFKA_ADVERTISED_PORT
value: "9092"
- name: GODEBUG
value: netdns=go
RobertDiebels (Sat, 26 May 2018 09:38:00 GMT):
Hey guys, just published a new version of https://www.npmjs.com/package/kubechain now supports chaincode and channels out of the box.
RobertDiebels (Sat, 26 May 2018 09:38:27 GMT):
Works on Windows (tested) should work on Linux (untested).
RobertDiebels (Sat, 26 May 2018 09:40:22 GMT):
I found a work around for the Docker container issue and having to reboot your nodes as mentioned in https://hackernoon.com/how-to-deploy-hyperledger-fabric-on-kubernetes-1-a2ceb3ada078 . All you have to do is supply the tool with the cluster you intend to run on. It than adds the Kube-dns IP correctly, without requiring a reboot.
Taffies (Mon, 28 May 2018 04:54:46 GMT):
hi! anyone successfully managed to enable TLS on kubernetes setup? :)
Taffies (Mon, 28 May 2018 04:57:10 GMT):
the other question is - do we need tls on kubernetes?
peter.li (Tue, 29 May 2018 00:31:38 GMT):
Has joined the channel.
alejandrolr (Tue, 29 May 2018 21:25:15 GMT):
Has joined the channel.
alejandrolr (Tue, 29 May 2018 21:32:26 GMT):
Hi all!! I'm working in developing a simple Fabric application using the Node SDK on Kubernetes. Firstly, I've deployed a custom kubernetes cluster in Google Kubernetes Engine and created a simple network using this repo from steps 4 and 5 (https://github.com/IBM/blockchain-network-on-kubernetes). Note: I skip the IBM Cloud Steps because I'm using my own cluster.
After the deployment, I'm able to perform invokes/queries inside one peer, without problem. However, if I use the Node SDK (https://github.com/hyperledger/fabric-samples/tree/release-1.1/fabcar) I'm only able to perform enroll/register and queries. The invoke requests fail. Do you have any idea?
```root@kubernetes-api-85df84499d-g8tgd:/api-v1.0# node enrollAdmin.js
Store path:/api-v1.0/hfc-key-store
Successfully enrolled admin user "admin"
Assigned the admin user to the fabric client ::{"name":"admin","mspid":"Org1MSP","roles":null,"affiliation":"","enrollmentSecret":"","enrollment":{"signingIdentity":"89258074fc553d792ea5d4685097c28352191e9b8587cbe80b649bd59ea6c877","identity":{"certificate":"-----BEGIN CERTIFICATE-----\nMIIB8DCCAZegAwIBAgIUNhnBDWYMEFpx0BAKwZmj7Xl6GkgwCgYIKoZIzj0EAwIw\nczELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcTDVNh\nbiBGcmFuY2lzY28xGTAXBgNVBAoTEG9yZzEuZXhhbXBsZS5jb20xHDAaBgNVBAMT\nE2NhLm9yZzEuZXhhbXBsZS5jb20wHhcNMTgwNTI5MTkxMDAwWhcNMTkwNDI4MDMx\nMDAwWjAQMQ4wDAYDVQQDEwVhZG1pbjBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IA\nBAnEGtTNf/ScMtwPF7ANl/kisWEpx0/6UgPvi41i7a6Oc1SCFCpXE0huz94xy9Y5\nhIZecX/WqBPezcOTzcCBm/WjbDBqMA4GA1UdDwEB/wQEAwICBDAMBgNVHRMBAf8E\nAjAAMB0GA1UdDgQWBBR43OWbfNXwL/1/EQz5+me+IYTr8zArBgNVHSMEJDAigCD/\nik1WxH34lDg4L9LpQ80ga5BBtglshohRRmQbAlsdUDAKBggqhkjOPQQDAgNHADBE\nAiAfV6y4giHpPsauLH2JDqDFDVrYbX2fP7dCDhG8tNPujQIgY1hMgYMp+Qzzvvc5\nY8sEenHVszjtbWbGvAwily0Nrh4=\n-----END CERTIFICATE-----\n"}}}
root@kubernetes-api-85df84499d-g8tgd:/api-v1.0# node registerUser.js
Store path:/api-v1.0/hfc-key-store
Successfully loaded admin from persistence
Successfully registered user1 - secret:pXTXFqjvnzri
Successfully enrolled member user "user1"
User1 was successfully registered and enrolled and is ready to intreact with the fabric network
root@kubernetes-api-85df84499d-g8tgd:/api-v1.0# node query.js
Store path:/api-v1.0/hfc-key-store
Successfully loaded user1 from persistence
Query has completed, checking results
Response is 60
root@kubernetes-api-85df84499d-g8tgd:/api-v1.0# node invoke.js
Store path:/api-v1.0/hfc-key-store
Successfully loaded user1 from persistence
Assigning transaction_id: 1b05a3be4f4752b7005d95ac6bceed22d827cea9ae7833c374a66866ead8916f
error: [Peer.js]: sendProposal - timed out after:45000
error: [client-utils.js]: sendPeersProposal - Promise is rejected: Error: REQUEST_TIMEOUT
at Timeout._onTimeout (/api-v1.0/node_modules/fabric-client/lib/Peer.js:117:19)
at ontimeout (timers.js:498:11)
at tryOnTimeout (timers.js:323:5)
at Timer.listOnTimeout (timers.js:290:5)
Transaction proposal was bad
Failed to send Proposal or receive valid response. Response null or status is not 200. exiting...
Failed to invoke successfully :: Error: Failed to send Proposal or receive valid response. Response null or status is not 200. exiting...```
alejandrolr (Tue, 29 May 2018 22:23:40 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=BCkWpC7hr6EBszyrw) @JayJong Hi JayJong, how do you configure your network? Because Im not able to invoke using the fabcar sdk sample... However im able to query
AlexanderZhovnuvaty (Wed, 30 May 2018 11:37:40 GMT):
Has joined the channel.
rogerwilcos (Wed, 30 May 2018 23:14:52 GMT):
Has joined the channel.
bayesian_thought (Thu, 31 May 2018 03:07:07 GMT):
Has joined the channel.
BBurgDave (Thu, 31 May 2018 20:29:47 GMT):
Has joined the channel.
JayJong (Sat, 02 Jun 2018 16:14:32 GMT):
@alejandrolr looking at ur error msg, it may be ur network speed of ur virtual machines(or wherever u are setting ur kubernetes cluster) are too slow so it times out after 45 secs
alejandrolr (Sat, 02 Jun 2018 20:06:43 GMT):
thank you!! I've fixed my mistake, the problem was that I was using a cluster in GKE of Container Optimized OS. I've change them to Ubuntu and now I'm able to perform invokes
alejandrolr (Sat, 02 Jun 2018 20:06:43 GMT):
thank you!! I've fixed my mistake, the problem was that I was using a cluster in GKE of Container Optimized OS images. I've change them to Ubuntu and now I'm able to perform invokes...
Aswath8687 (Mon, 04 Jun 2018 17:53:44 GMT):
Has joined the channel.
karthik.ir (Tue, 05 Jun 2018 08:03:44 GMT):
Has joined the channel.
IsaacWong (Wed, 06 Jun 2018 03:09:53 GMT):
Has joined the channel.
vidor (Wed, 06 Jun 2018 09:03:13 GMT):
Has joined the channel.
koineramitranjan (Wed, 06 Jun 2018 11:08:53 GMT):
Has joined the channel.
koineramitranjan (Wed, 06 Jun 2018 11:09:01 GMT):
aws
pd93 (Wed, 06 Jun 2018 12:53:05 GMT):
Has joined the channel.
DineshPrabhu (Thu, 07 Jun 2018 07:55:53 GMT):
Has joined the channel.
smallX (Thu, 07 Jun 2018 08:32:05 GMT):
Has joined the channel.
dklesev (Thu, 07 Jun 2018 09:31:11 GMT):
Has joined the channel.
PawelD (Thu, 07 Jun 2018 19:33:47 GMT):
Has joined the channel.
wangrangli (Fri, 08 Jun 2018 05:30:55 GMT):
Has joined the channel.
abraham (Fri, 08 Jun 2018 05:42:22 GMT):
Has joined the channel.
goranovic (Sat, 09 Jun 2018 11:07:05 GMT):
Has joined the channel.
nagap (Mon, 11 Jun 2018 15:26:37 GMT):
Has joined the channel.
smallX (Tue, 12 Jun 2018 03:58:35 GMT):
hi, guys. i followed this link [https://medium.com/@zhanghenry/how-to-deploy-hyperledger-fabric-on-kubernetes-2-751abf44c807](https://medium.com/@zhanghenry/how-to-deploy-hyperledger-fabric-on-kubernetes-2-751abf44c807) to deploy kubernetes cluster. but there are errors
```
The connection to the server localhost:8080 was refused - did you specify the right host or port?
The connection to the server localhost:8080 was refused - did you specify the right host or port?
The connection to the server localhost:8080 was refused - did you specify the right host or port?
The connection to the server localhost:8080 was refused - did you specify the right host or port?
The connection to the server localhost:8080 was refused - did you specify the right host or port?
The connection to the server localhost:8080 was refused - did you specify the right host or port?
The connection to the server localhost:8080 was refused - did you specify the right host or port?
```
when i run `python3.5 transform/run.py`
smallX (Tue, 12 Jun 2018 03:59:17 GMT):
```
root@org2:~/k8s# kubectl get nodes
NAME STATUS ROLES AGE VERSION
order1 Ready
smallX (Tue, 12 Jun 2018 04:03:08 GMT):
```
kubectl get pod -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE
coredns-7997f8864c-d2j8m 1/1 Running 0 5d xx.xxx.0.4 org2
coredns-7997f8864c-rqrw4 1/1 Running 0 5d xx.xxx.1.2 order1
etcd-org2 1/1 Running 0 5d xxx.xxx36.92 org2
kube-apiserver-org2 1/1 Running 0 5d xxx.xxx.36.92 org2
kube-controller-manager-org2 1/1 Running 0 5d xxx.xxx.36.92 org2
kube-flannel-ds-9s2pt 1/1 Running 0 5d xxx.xx.217.103 order3
kube-flannel-ds-jt6vd 1/1 Running 0 5d xxx.xxx.36.92 org2
kube-flannel-ds-jtfnj 1/1 Running 0 5d xxx.xx.173.39 order4
kube-flannel-ds-lqnms 1/1 Running 1 5d xxx.xx.xxx.201 order2
kube-flannel-ds-wwvxm 1/1 Running 0 5d xxx.xx.xxx.46 order1
kube-proxy-2kshk 1/1 Running 0 5d xxx.28.xxx.46 order1
kube-proxy-97t9w 1/1 Running 0 5d xxx.28.xxx.103 order3
kube-proxy-nms8b 1/1 Running 0 5d xxx.28.xxx.39 order4
kube-proxy-qdfrh 1/1 Running 0 5d xxx.129.xx.xx org2
kube-proxy-wzrg2 1/1 Running 0 5d xxx.28.xx.xxx order2
kube-scheduler-org2 1/1 Running 0 5d xxx.129.xx.xx org2
```
RobertDiebels (Tue, 12 Jun 2018 07:37:37 GMT):
@smallX You could give my CLI a go [ https://www.npmjs.com/package/kubechain ] if you want.
RobertDiebels (Tue, 12 Jun 2018 07:38:07 GMT):
I based my design on the tutorial you linked above.
RobertDiebels (Tue, 12 Jun 2018 07:39:02 GMT):
Right now the fabric version is fixed. So if you want to change the version you'll have to make a tiny change in the code: https://github.com/kubechain/kubechain
titoe (Tue, 12 Jun 2018 09:46:38 GMT):
Has joined the channel.
CarlitoIBM (Tue, 12 Jun 2018 14:14:23 GMT):
Has joined the channel.
smallX (Wed, 13 Jun 2018 03:13:07 GMT):
@RobertDiebels ths. i'll study the link. but now i prefer to resolve this problem.
david_dornseifer (Wed, 13 Jun 2018 15:33:41 GMT):
Has joined the channel.
Nghiadt (Wed, 13 Jun 2018 17:56:13 GMT):
Has joined the channel.
JayJong (Thu, 14 Jun 2018 09:55:58 GMT):
where are u running kubernetes? ur com or some vm? i would think the error is due to port 8080 does not allow inbound connection
nabilm (Thu, 14 Jun 2018 12:25:29 GMT):
Has joined the channel.
paulananth (Thu, 14 Jun 2018 16:22:11 GMT):
Has joined the channel.
SeamonJia (Mon, 18 Jun 2018 09:58:21 GMT):
Has joined the channel.
alexvicegrab (Tue, 19 Jun 2018 14:23:02 GMT):
At AID:Tech, @nicolapaoli and I are open-sourcing some Helm Charts to deploy Fabric on Kubernetes, stay tuned, and feel free to contribute to them once they are approved:
https://github.com/kubernetes/charts/pull/6181
https://github.com/kubernetes/charts/pull/6165
https://github.com/kubernetes/charts/pull/6163
https://github.com/kubernetes/charts/pull/6144
sunilbalaguragi (Tue, 19 Jun 2018 16:57:34 GMT):
Has joined the channel.
sunilbalaguragi (Tue, 19 Jun 2018 16:58:14 GMT):
are there any instructions to install hyperledger on minikube on windows platform
Nghiadt (Tue, 19 Jun 2018 17:20:48 GMT):
MultiOrderers-Kafka-Kubernetes
Nghiadt (Tue, 19 Jun 2018 17:20:48 GMT):
MultiOrderers-Kafka-Kubernetes
Nghiadt (Tue, 19 Jun 2018 17:20:48 GMT):
MultiOrderers-Kafka-Kubernetes
Nghiadt (Tue, 19 Jun 2018 17:21:15 GMT):
Hi all, I'm trying to build MULTIPLE orderers on Kafka on Kubernetes. I've build 2 ORGs with 2 PEERs per 1 ORG, 3 Zoo, 4 Kaffa and 2 Orderer and I think the procress looks like this:
- Each ORG will choose a PEER that connects to 1 Orderer to create a channel calling _mychannel_. In my case, I configured PEER0-ORG1 to Orderer0 and PEER0-ORG2 to Orderer1
- Then, each PEER will join the channel via the Orderer corresponding to its own ORG
- Finally, updating the anchor peer for ORGs
But, I'm in trouble at step one when trying to create channel :) Anyone can help me, please?
Nghiadt (Tue, 19 Jun 2018 17:21:15 GMT):
Hi all, I'm trying to build MULTIPLE orderers on Kafka on Kubernetes. I've build *2 ORGs with 2 PEERs per 1 ORG, 3 Zoo, 4 Kaffa and 2 Orderer* and I think the procress looks like this:
- Each ORG will choose a PEER that connects to 1 Orderer to create a channel calling *mychannel*. In my case, I configured PEER0-ORG1 to Orderer0 and PEER0-ORG2 to Orderer1
- Then, each PEER will join the channel via the Orderer corresponding to its own ORG
- Finally, updating the anchor peer for ORGs
But, I'm in trouble at step one when trying to create channel :) Anyone can help me, please?
Nghiadt (Tue, 19 Jun 2018 17:25:49 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=BfSW9poybKM6Ev45d) @sukrit.handa@gmail.com Hi Sukrit, Did you used multiple orderers or only one?
RobertDiebels (Wed, 20 Jun 2018 07:26:16 GMT):
@alexvicegrab Nice! I've been waiting for someone to do this. Feel free to take a look at my tool if you run into some issues: https://www.npmjs.com/package/kubechain . It's fully functional for 1.0.x. I use configMaps/secrets to mount certificates in my containers and copy them into a PV instead of using NFS or some other form of storage. I did so because it would allow certificate updates to be processed immediately in 1.1.x. Since 1.0.x doesn't follow sym-links for certificates. Either way hope you find something useful in there :D /
alexvicegrab (Wed, 20 Jun 2018 10:09:01 GMT):
@RobertDiebels, cool, I did check out your tool early on, awesome that you are also doing work on this front.
We also use configMaps and secrets for a number of the certificates (the ones relating to Organisations). Hopefully we can learn from each other and improve both tools :D
RobertDiebels (Wed, 20 Jun 2018 10:28:26 GMT):
@alexvicegrab yea that would be great! I'd love to chuck out the kubernetes resource generation part and just feed the configuration into the helm files. That would save me a boatload of maintenance :P
alexvicegrab (Wed, 20 Jun 2018 10:29:50 GMT):
Cool, feel free to use and contribute to my charts, which are created for the resource generation, but not for the coordination between them. That might be a good division of labour for collaborating :)
alexvicegrab (Wed, 20 Jun 2018 10:30:33 GMT):
If you like, we can do a call sometime and I can show you how I have set them up to work, and you might see how they fit with kubechain and whether you can suggest improvements :)
RobertDiebels (Wed, 20 Jun 2018 10:33:01 GMT):
Yea that would be great. I don't know when I'm free to do so, since I'm working on integrating Hyperledger Caliper as well.
RobertDiebels (Wed, 20 Jun 2018 10:33:01 GMT):
Yea that would be great. I don't know when I'm free to do so, since I'm working on integrating Hyperledger Caliper into Kubernetes as well.
RobertDiebels (Wed, 20 Jun 2018 10:34:04 GMT):
I'll send you a PM when I'm available :D :thumbsup:
alexvicegrab (Wed, 20 Jun 2018 10:36:02 GMT):
Cool, we'll find a day/time when we both are available, thanks Robert!
sunilbalaguragi (Wed, 20 Jun 2018 12:02:04 GMT):
@RobertDiebels, I am getting this error ? Which cluster context do you want to use? minikube
Setting up kubernetes-client
Getting kube-dns service IP-address
[PEER-ORGANISATION]: org1
Creating configuration directories
Unable to create complete fabric configuration.
Reason: TypeError: Cannot read property 'forEach' of undefined
at PeerOrganization.createChainCodes (...\\AppData\Roaming\npm\node_modules\kubechain\src\main\lib\blockchains\fabric\adapters\minikube\organizations\peer.js:154:20)
RobertDiebels (Wed, 20 Jun 2018 12:24:09 GMT):
@sunilbalaguragi Add a chaincodes: [] I still need to add default options see: https://github.com/kubechain/kubechain/blob/master/docs/tutorials/fabric/configuration-samples/kubechain/chaincode-and-channel.kubechain.config.js
Nghiadt (Wed, 20 Jun 2018 14:47:44 GMT):
kàka
Nghiadt (Wed, 20 Jun 2018 17:05:06 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=L5Ho8LtMH8Jh8DwvX) @vanitas92 Hi @vanitas92 Are you using multiple or single orderer? I'm being in trouble with multiple orderers - kafka because of cannot lookup the orderers domain name
sunilbalaguragi (Thu, 21 Jun 2018 00:33:24 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=q3yXyB3nrgwkGG8kQ) @RobertDiebels - Please let me know, where should I add the chaincodes. I copied kubechain.config.js, but did not help
RobertDiebels (Thu, 21 Jun 2018 07:34:45 GMT):
@sunilbalaguragi try version 0.6.2 instead of 0.6.3 if you will.
RobertDiebels (Thu, 21 Jun 2018 07:40:17 GMT):
I just checked https://github.com/kubechain/kubechain/blob/master/src/main/ts/blockchains/fabric/adapters/minikube/organizations/peer.ts#L154 which is about chaincodes. The file I linked above should contain a property called `options` with an object and another property `chaincodes` which should hold an empty Array.
vanitas92 (Thu, 21 Jun 2018 08:21:45 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=goqtvz5W5M5qnCkLi) @Nghiadt I'm using a single orderer. I have not try the solution everyone is commenting here of adding the GODEBUG variable at the yaml but i think it should work. I am currently using 4 deployments for 4 kafka brokers
NurDerFSV (Thu, 21 Jun 2018 09:17:47 GMT):
Has joined the channel.
NurDerFSV (Thu, 21 Jun 2018 09:21:59 GMT):
hello, does anyone have any experiences with dcos or marathon instead kubernetes ?
Nghiadt (Thu, 21 Jun 2018 09:28:49 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=72RAnv9ENiSqjqQ2m) @vanitas92 I'll try this one. Btw, I've just seen your messages which is about the instantiation chaincodes on kubernetes. Have you done yet? I pass it, maybe i can help
Nghiadt (Thu, 21 Jun 2018 09:28:49 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=72RAnv9ENiSqjqQ2m) @vanitas92 I'll try this one. Btw, I've just seen your message which is about the instantiation chaincodes on kubernetes. Have you done yet? I pass it, maybe i can help
Nghiadt (Thu, 21 Jun 2018 09:33:09 GMT):
Orderers-Kafkas-Zoos
Nghiadt (Thu, 21 Jun 2018 09:33:11 GMT):
Hi everyone, My Kafka-Zoo cluster cannot find endpoint orderers. I think, maybe Kube-DNS cannot lookup the orderer's domain name.
How should I do now?
vanitas92 (Thu, 21 Jun 2018 17:19:36 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=WJ5SzYebqR4apuxXH) @Nghiadt Ive tried one way but you have to enable dev mode on your chaincodes to do so. I do not reccomend this approach and its not suitable for dynamic upgrades or instantiation of new chaincodes. I wait until the dev team at hyperledger comes a good way of adding this feature. I follow this issue in jira which they discuss a way of approaching this but since february there is no activity: https://jira.hyperledger.org/browse/FAB-7406
Nghiadt (Thu, 21 Jun 2018 23:14:30 GMT):
@vanitas92 I've just added two vars env. You can try it
`- name: CORE_PEER_TLS_SERVERHOSTOVERRIDE # The server name use this name to verify the hostname returned by TLS handshake`
`value:
Nghiadt (Thu, 21 Jun 2018 23:16:46 GMT):
@vanitas92 I finded this solution from another one in this channel. You can search histories chat to check again.
sunilbalaguragi (Thu, 21 Jun 2018 23:24:25 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=H43dnyfrDeaAx4BGG) @RobertDiebels Robert, Thanks for looking into this, I checked peer.ts file it looks good. In the peer*.js*, I replaced this line var chaincodes = this.options.get('$.options.chaincodes'); with an empty array, then it did not generate any error.
Nghiadt (Fri, 22 Jun 2018 00:51:25 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=bZpGrwHj979Gh4vQK) @joaquimpedrooliveira are you using multiple or one orderer?
Nghiadt (Fri, 22 Jun 2018 01:12:00 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=GCjnyFaPLFjLzujHJ) @vanitas92 I solved this. Because I configured Kafka cluster (Kafka & Zoo) in namespace different from the Orderer but I do not point it out :) Then i decided letting them (Orderer, Kafkas, Zoos) in the same namespace. So, it works!
Nghiadt (Fri, 22 Jun 2018 01:12:00 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=GCjnyFaPLFjLzujHJ) @vanitas92 I solved this. Because I configured Kafka cluster (Kafka & Zoo) in a namespace different from the Orderer's one but I do not point it out :) Then i decided letting them (Orderer, Kafkas, Zoos) in the same namespace. So, it works!
vanitas92 (Fri, 22 Jun 2018 07:26:21 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=j3hYqM9HtTSzjoeob) @Nghiadt Ok but you have to link the docker socket with kubernetes somehow, aprt from adding this variables what else have you done in order to make it work?
vanitas92 (Fri, 22 Jun 2018 07:26:21 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=j3hYqM9HtTSzjoeob) @Nghiadt Ok but you have to link the docker socket with kubernetes somehow, apart from adding these variables what else have you done in order to make it work?
RobertDiebels (Fri, 22 Jun 2018 08:18:47 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=XXtHpE7WGA5ivDAjz) @sunilbalaguragi That's not exactly what I meant. I meant you should alter the kubechain.config.js file so that the chaincodes property is an empty array. If you got it working though it's fine.
govinda-attal (Fri, 22 Jun 2018 10:40:31 GMT):
Hello All, Just a quick question ??? Having Fabric on Kubernetes is possible and but last time I tried for the settings I used, I was able install and instantiate chaincode and transact with same. But the container for chaincode was not visible to Kubernetes and so not managed by it... Is there a way to have chaincode and their versions now visible and managed by Kubernetes ???
nvxtien (Fri, 22 Jun 2018 11:26:55 GMT):
Has joined the channel.
Lakshmipadmaja (Fri, 22 Jun 2018 12:18:19 GMT):
Has joined the channel.
MonnyClara (Fri, 22 Jun 2018 12:33:02 GMT):
@govinda-attal To my knowledge not yet ! But I know that some issues have been opened in Jira about this.
As I remember, the idea is about being able to tell Fabric if the chaincode must be instantiated on the docker daemon or on K8s.
MonnyClara (Fri, 22 Jun 2018 12:34:18 GMT):
https://jira.hyperledger.org/browse/FAB-5578?jql=text%20~%20%22kubernetes%22
Nghiadt (Sat, 23 Jun 2018 00:51:58 GMT):
@vanitas92 Nope, Just 2 options. Your chaincode cannot connect again to its peer because Docker DNS cannot reverse the peer IP from peer hostname. So, you have 2 options, configure docker host to find peer IP from its hostname or expose its IP.
In my case, I choose the second one. Setting
`CORE_PEER_ADDRESSAUTODETECT=true` the peer will automate expose its real IP for us instead of its hostname. And you use `CORE_PEER_TLS_SERVERHOSTOVERRIDE='peer-host-name'` for your TLS option.
Nghiadt (Sat, 23 Jun 2018 00:51:58 GMT):
@vanitas92 Nope, Just 2 options. Your chaincode cannot connect again to its peer because Docker DNS cannot reverse the peer IP from peer hostname. So, you have 2 options, configure docker host to find peer IP from its hostname or expose its IP from the beginning.
In my case, I choose the second one. Setting
`CORE_PEER_ADDRESSAUTODETECT=true` the peer will automate expose its real IP for us instead of its hostname. And you use `CORE_PEER_TLS_SERVERHOSTOVERRIDE='peer-host-name'` for your TLS option.
rudimk (Sat, 23 Jun 2018 04:15:23 GMT):
Has joined the channel.
sureshtedla (Sat, 23 Jun 2018 23:30:03 GMT):
Has joined the channel.
govinda-attal (Sun, 24 Jun 2018 18:49:13 GMT):
@MonnyClara Thank you this helps our team to decide for our current deployment options. 😊 Good day!
DavidPark (Tue, 26 Jun 2018 22:57:45 GMT):
Has joined the channel.
RobertDiebels (Wed, 27 Jun 2018 08:16:14 GMT):
Has left the channel.
axel (Wed, 27 Jun 2018 15:00:17 GMT):
Has joined the channel.
bejoypg (Thu, 28 Jun 2018 08:28:30 GMT):
Has joined the channel.
vinayakkumar (Fri, 29 Jun 2018 11:00:54 GMT):
Has joined the channel.
Nghiadt (Fri, 29 Jun 2018 17:01:57 GMT):
Kafka-Orderer-Error.png
Nghiadt (Fri, 29 Jun 2018 17:02:39 GMT):
I'm building Kafka on K8s and I've gotten this Error when trying to create channel.
Nghiadt (Fri, 29 Jun 2018 17:04:00 GMT):
Has anyone gotten it?
SDive (Sat, 30 Jun 2018 14:17:57 GMT):
Has joined the channel.
Othman.Darwish (Sun, 01 Jul 2018 09:27:22 GMT):
Hi all, anyone managed successfully running kafka orderer/zk on k8s, please share scripts
altairlee (Mon, 02 Jul 2018 08:07:58 GMT):
Has joined the channel.
futurama92 (Mon, 02 Jul 2018 09:31:47 GMT):
Has joined the channel.
n1zyz (Mon, 02 Jul 2018 12:28:42 GMT):
Has joined the channel.
Taffies (Tue, 03 Jul 2018 07:35:53 GMT):
hi! anyone used SSL with kafka on kubernetes before?
edevil (Tue, 03 Jul 2018 10:58:52 GMT):
Has joined the channel.
grice_32 (Tue, 03 Jul 2018 13:07:03 GMT):
Has left the channel.
sudhir.kumawat (Tue, 03 Jul 2018 15:30:55 GMT):
Has joined the channel.
sudhir.kumawat (Tue, 03 Jul 2018 15:33:23 GMT):
Hi i am trying to deploy business network archive(.bna) to hyperledger network setup on kubernetes and getting error when compose network install command using to install chaincode on fabric network Error; Unknown chaincodeType: NODE
Taffies (Wed, 04 Jul 2018 07:24:40 GMT):
hi! how do i switch over to another peer inside the CLI?
Taffies (Wed, 04 Jul 2018 07:24:52 GMT):
i want to the peer1 to the channel from the CLI but i'm unable to do so
Taffies (Wed, 04 Jul 2018 07:24:52 GMT):
i want to join the peer1 to the channel from the CLI but i'm unable to do so
giniz (Thu, 05 Jul 2018 09:23:03 GMT):
Has joined the channel.
ongkhaiwei (Thu, 05 Jul 2018 13:50:32 GMT):
Has joined the channel.
anzalbeg (Thu, 05 Jul 2018 14:53:55 GMT):
Hi All Just need one clarification on fabric-ca example in fabric-sample folder. I used fabric-ca example to create certificates via fabric Certificate Authority. After successfully running it, 1 orderer, 1 ca, 2 peers and 1 chaincode containers are started successfully. My Question is that Is it possible to deploy all this container services to kubernetes using *KOMPOSE *tool by converting the docker-compose.yaml file to kubernetes resources??
anzalbeg (Thu, 05 Jul 2018 14:55:30 GMT):
Can anyone have anything to suggest?
merth (Thu, 05 Jul 2018 18:52:14 GMT):
Has joined the channel.
thakurnikk (Mon, 09 Jul 2018 06:17:59 GMT):
Has joined the channel.
thakurnikk (Mon, 09 Jul 2018 11:30:47 GMT):
i am trying to setup hyperledger-fabric example using kubernetes in my local ubuntu machine
thakurnikk (Mon, 09 Jul 2018 11:32:02 GMT):
Screenshot from 2018-07-09 16-58-52.png
thakurnikk (Mon, 09 Jul 2018 11:34:11 GMT):
i am using http://www.think-foundry.com/deploy-hyperledger-fabric-on-kubernetes-part-2/ for setup, when i use the kubectl command to describe the pod it shows some problem with the docker? any body k
thakurnikk (Mon, 09 Jul 2018 11:34:11 GMT):
i am using http://www.think-foundry.com/deploy-hyperledger-fabric-on-kubernetes-part-2/ for setup, when i use the kubectl command to describe the pod it shows some problem with the docker? does anyone knows the solution
thakurnikk (Mon, 09 Jul 2018 11:34:11 GMT):
i am using http://www.think-foundry.com/deploy-hyperledger-fabric-on-kubernetes-part-2/ for setup, when i use the kubectl command to describe the pod it shows some problem with the docker? does anyone knows the solution @yacovm @jyellick
yacovm (Mon, 09 Jul 2018 11:45:09 GMT):
why are you tagging me? :/
baohua (Tue, 10 Jul 2018 03:09:14 GMT):
Has left the channel.
EricYang (Tue, 10 Jul 2018 04:27:16 GMT):
Has joined the channel.
Luke_Chen (Tue, 10 Jul 2018 05:47:59 GMT):
@thakurnikk That article is little bit out of date.
Luke_Chen (Tue, 10 Jul 2018 05:48:06 GMT):
use this on instead.
Luke_Chen (Tue, 10 Jul 2018 05:48:06 GMT):
use this one instead.
thakurnikk (Tue, 10 Jul 2018 05:49:44 GMT):
which one @Luke_Chen ?
thakurnikk (Tue, 10 Jul 2018 05:49:44 GMT):
okay @Luke_Chen, Thaknyou :)
Luke_Chen (Tue, 10 Jul 2018 05:50:01 GMT):
https://labs.vmware.com/flings/blockchain-on-kubernetes
Luke_Chen (Tue, 10 Jul 2018 05:50:11 GMT):
skip the part of setting up kubernetes
thakurnikk (Tue, 10 Jul 2018 05:52:02 GMT):
okay @Luke_Chen, Thaknyou :)
sekipaolo (Tue, 10 Jul 2018 15:11:08 GMT):
Has joined the channel.
jayeshjawale95 (Wed, 11 Jul 2018 07:30:36 GMT):
Has joined the channel.
sudhir.kumawat (Thu, 12 Jul 2018 08:11:09 GMT):
Hello everyone I am trying to install and start business network over hyperledger fabric running on kubernetes with composer and getting error Failed to generate platform-specific docker build: Error returned from build: 1 "npm ERR! code EAI_AGAIN
npm ERR! errno EAI_AGAIN
npm ERR! request to https://registry.npmjs.org/composer-runtime-hlfv1 failed, reason: getaddrinfo EAI_AGAIN registry.npmjs.org:443
sudhir.kumawat (Thu, 12 Jul 2018 08:11:28 GMT):
Can anyone help me to resolve this issue.
sudhir.kumawat (Thu, 12 Jul 2018 08:12:17 GMT):
I am using composer version 0.19.12 HF version 1.1.0 Node version 8.10.0
leolustig (Thu, 12 Jul 2018 14:16:04 GMT):
Has joined the channel.
gsgx (Thu, 12 Jul 2018 15:00:47 GMT):
Has joined the channel.
BlockchainBusiness (Mon, 16 Jul 2018 02:33:41 GMT):
Has joined the channel.
tejpowar (Mon, 16 Jul 2018 10:02:15 GMT):
Has joined the channel.
vkblue (Mon, 16 Jul 2018 23:29:02 GMT):
Has joined the channel.
alek (Tue, 17 Jul 2018 10:07:00 GMT):
Has joined the channel.
alejandrolr (Tue, 17 Jul 2018 10:10:59 GMT):
Hello everybody, I'm trying to set up fabric 1.2 on kubernetes, has someone successfully got it?
alejandrolr (Tue, 17 Jul 2018 10:11:43 GMT):
with 1.0 it works fine but I want to upgrade my components
sudhir.kumawat (Wed, 18 Jul 2018 10:42:49 GMT):
hi everyone i want to change network mode from bridge(default) to host in chaincode containers spin up through composer network start command.Anyone can help out. This is the specific case where mode is not changing. however i am able to change network mode of those chaincode containers which are instantiate with peer chaincode instantiate command
vanitas92 (Wed, 18 Jul 2018 12:13:10 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=eFn9bLjzopf9QLszR) @sudhir.kumawat You can set the following environment variable in the yamls of deployment:
- name: CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE
value: host
sudhir.kumawat (Wed, 18 Jul 2018 12:23:02 GMT):
Hi @vanitas92 , I have used this in peers yaml and chaincode containers are up with host n/w. But i want to change network setting of those chaincode containers which are up with composer network start
javrevasandeep (Wed, 18 Jul 2018 17:50:49 GMT):
Has joined the channel.
javrevasandeep (Wed, 18 Jul 2018 18:15:54 GMT):
@Luke_Chen I am facing this issue while creating channel
grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 192.168.99.100:7050
newthinker (Thu, 19 Jul 2018 01:58:16 GMT):
Has joined the channel.
Luke_Chen (Thu, 19 Jul 2018 12:52:11 GMT):
@javrevasandeep Any detail logs can be tracked ?
iwinoto (Thu, 19 Jul 2018 13:02:24 GMT):
Has joined the channel.
rajanashutosh (Thu, 19 Jul 2018 14:29:18 GMT):
Has joined the channel.
amongv587 (Thu, 19 Jul 2018 15:07:31 GMT):
Has joined the channel.
javrevasandeep (Thu, 19 Jul 2018 15:39:54 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=AHnNA5mMs9jpui4LX) @Luke_Chen Below given is my helm value.yaml file
# Default values for fabric-artifacts.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
fabricTag: &fabricTag x86_64-1.0.5
replicaCount: 1
image:
repository: nginx
tag: stable
pullPolicy: IfNotPresent
service:
name: nginx
type: ClusterIP
externalPort: 80
internalPort: 80nam
fabricOrderer:
addr: 192.168.99.100
orgDomain: org1.example.com
replicas: 1
image: hyperledger/fabric-orderer
tag: *fabricTag
orgDomain: example.com
consensusType: solo
msp: OrdererMSP
fabricOrg1:
domain: org1.example.com
msp: Org1MSP
fabricOrg2:
domain: org2.example.com
msp: Org2MSP
fabricCa1:
keyStoreHash: c4d6886fb9125ae7c69acbfcb965eeed7b7d39c0ce65293b369175aa3635cdaf
image: hyperledger/fabric-ca
tag: *fabricTag
fabricCa2:
keyStoreHash: 50802fade11ec16c31e4ca4979137f8d83750b84986dcb7e100cb0f21d3c2952
image: hyperledger/fabric-ca
tag: *fabricTag
fabricPeer:
image: hyperledger/fabric-peer
tag: *fabricTag
ingress:
enabled: false
# Used to create an Ingress record.
hosts:
- example.com
addr: 192.168.99.100
annotations:
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
tls:
# Secrets must be manually created in the namespace.
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
resources: {}
Swaroop.Kondru (Sat, 21 Jul 2018 19:12:35 GMT):
Has joined the channel.
maheshwarishikha (Sun, 22 Jul 2018 06:02:20 GMT):
Has joined the channel.
rajanashutosh (Sun, 22 Jul 2018 07:58:20 GMT):
Hi all, I have done setup off hyper-ledger fabric using docker compose, I would like to know while setting up hyperledger fabric using kubernetes what would be the difference. Also if you could provide me some tutorials to follow would be of great help
maheshwarishikha (Sun, 22 Jul 2018 08:14:01 GMT):
Hi all, I am upgrading the hyperledger fabric network on Kubernetes to 1.1 from 1.0. All things worked fine till chaincode install but chaincode instantiation is failing with error:
"Error: Error endorsing chaincode: rpc error: code = Unknown desc = failed to execute transaction: timeout expired while executing transaction"
I have tried setting 'CORE_CHAINCODE_STARTUPTIMEOUT' but no luck. Can anyone give pointers to resolve this error. TIA.
rajanashutosh (Mon, 23 Jul 2018 06:56:34 GMT):
In a multi-host environment which is better going with Kubernetes or Docker Swarm.?
rajanashutosh (Mon, 23 Jul 2018 07:06:08 GMT):
Also, if we are going ahead with Kubernetes which networking model is best suited, Flannel or Docker Overlay network. ?
anzalbeg (Mon, 23 Jul 2018 12:07:19 GMT):
Hello guys. I am facing issue while instantiating the chaincode on google kubernetes engine
anzalbeg (Mon, 23 Jul 2018 12:10:02 GMT):
below given is the cli container logs
anzalbeg (Mon, 23 Jul 2018 12:10:32 GMT):
CORE_CHAINCODE_LOGGING_FORMAT=%{color}%{time:2006-01-02 15:04:05.000 MST} [%{module}] %{shortfunc} -> %{level:.4s} %{id:03x}%{color:reset} %{message}
2018-07-23 11:57:42.109 UTC [container] lockContainer -> DEBU 3c7 waiting for container(dev-peer1.org1.example.com-mycc-1.0) lock
2018-07-23 11:57:42.109 UTC [container] lockContainer -> DEBU 3c8 got container (dev-peer1.org1.example.com-mycc-1.0) lock
2018-07-23 11:57:42.109 UTC [dockercontroller] Start -> DEBU 3c9 Cleanup container dev-peer1.org1.example.com-mycc-1.0
2018-07-23 11:57:42.111 UTC [dockercontroller] stopInternal -> DEBU 3ca Stop container dev-peer1.org1.example.com-mycc-1.0(Container not running: dev-peer1.org1.example.com-mycc-1.0)
2018-07-23 11:57:42.112 UTC [dockercontroller] stopInternal -> DEBU 3cb Kill container dev-peer1.org1.example.com-mycc-1.0 (API error (500): {"message":"Cannot kill container dev-peer1.org1.example.com-mycc-1.0: Container df448a5162f54077d65788582ac8df52bc658166df75684bf4467411dde4ae17 is not running"}
)
2018-07-23 11:57:42.123 UTC [dockercontroller] stopInternal -> DEBU 3cc Removed container dev-peer1.org1.example.com-mycc-1.0
2018-07-23 11:57:42.123 UTC [dockercontroller] Start -> DEBU 3cd Start container dev-peer1.org1.example.com-mycc-1.0
2018-07-23 11:57:42.123 UTC [dockercontroller] getDockerHostConfig -> DEBU 3ce docker container hostconfig NetworkMode: host
2018-07-23 11:57:42.124 UTC [dockercontroller] createContainer -> DEBU 3cf Create container: dev-peer1.org1.example.com-mycc-1.0
2018-07-23 11:57:42.170 UTC [dockercontroller] createContainer -> DEBU 3d0 Created container: dev-peer1.org1.example.com-mycc-1.0-cd123150154e6bf2df7ce682e0b1bcbea40499416f37a6da3aae14c4eb51b08d
2018-07-23 11:57:42.396 UTC [dockercontroller] Start -> DEBU 3d1 Started container dev-peer1.org1.example.com-mycc-1.0
2018-07-23 11:57:42.396 UTC [container] unlockContainer -> DEBU 3d2 container lock deleted(dev-peer1.org1.example.com-mycc-1.0)
2018-07-23 11:57:42.421 UTC [chaincode] HandleChaincodeStream -> DEBU 3d3 Current context deadline = 0001-01-01 00:00:00 +0000 UTC, ok = false
2018-07-23 11:57:42.422 UTC [chaincode] processStream -> DEBU 3d4 []Received message REGISTER from shim
2018-07-23 11:57:42.422 UTC [chaincode] handleMessage -> DEBU 3d5 []Fabric side Handling ChaincodeMessage of type: REGISTER in state created
2018-07-23 11:57:42.422 UTC [chaincode] beforeRegisterEvent -> DEBU 3d6 Received REGISTER in state created
2018-07-23 11:57:42.422 UTC [chaincode] registerHandler -> DEBU 3d7 registered handler complete for chaincode mycc:1.0
2018-07-23 11:57:42.422 UTC [chaincode] beforeRegisterEvent -> DEBU 3d8 Got REGISTER for chaincodeID = name:"mycc:1.0" , sending back REGISTERED
2018-07-23 11:57:42.422 UTC [chaincode] notifyDuringStartup -> DEBU 3d9 Notifying during startup
2018-07-23 11:57:42.422 UTC [chaincode] func1 -> DEBU 3da chaincode mycc:1.0 launch seq completed
2018-07-23 11:57:42.423 UTC [chaincode] ready -> DEBU 3db sending READY
2018-07-23 11:57:42.423 UTC [chaincode] setChaincodeProposal -> DEBU 3dc Setting chaincode proposal context...
2018-07-23 11:57:42.423 UTC [chaincode] setChaincodeProposal -> DEBU 3dd Proposal different from nil. Creating chaincode proposal context...
2018-07-23 11:57:42.423 UTC [chaincode] processStream -> DEBU 3de [8fae273c]Move state message READY
2018-07-23 11:57:42.423 UTC [chaincode] handleMessage -> DEBU 3df [8fae273c]Fabric side Handling ChaincodeMessage of type: READY in state established
2018-07-23 11:57:42.423 UTC [chaincode] enterReadyState -> DEBU 3e0 [8fae273c]Entered state ready
2018-07-23 11:57:42.423 UTC [chaincode] notify -> DEBU 3e1 notifying Txid:8fae273cee64d4743ca181f848f796a9b2649869bccf5647fa335e20c59ab850, channelID:mychannel
2018-07-23 11:57:42.423 UTC [chaincode] processStream -> DEBU 3e2 [8fae273c]sending state message READY
2018-07-23 11:57:42.423 UTC [chaincode] Launch -> DEBU 3e3 sending init completed
2018-07-23 11:57:42.423 UTC [chaincode] Launch -> DEBU 3e4 LaunchChaincode complete
2018-07-23 11:57:42.423 UTC [chaincode] Execute -> DEBU 3e5 Entry
2018-07-23 11:57:42.423 UTC [chaincode] Execute -> DEBU 3e6 chaincode canonical name: mycc:1.0
anzalbeg (Mon, 23 Jul 2018 12:10:55 GMT):
2018-07-23 11:57:42.423 UTC [chaincode] sendExecuteMessage -> DEBU 3e7 [8fae273c]Inside sendExecuteMessage. Message INIT
2018-07-23 11:57:42.424 UTC [chaincode] setChaincodeProposal -> DEBU 3e8 Setting chaincode proposal context...
2018-07-23 11:57:42.424 UTC [chaincode] setChaincodeProposal -> DEBU 3e9 Proposal different from nil. Creating chaincode proposal context...
2018-07-23 11:57:42.424 UTC [chaincode] sendExecuteMessage -> DEBU 3ea [8fae273c]sendExecuteMsg trigger event INIT
2018-07-23 11:57:42.424 UTC [chaincode] processStream -> DEBU 3eb [8fae273c]Move state message INIT
2018-07-23 11:57:42.424 UTC [chaincode] handleMessage -> DEBU 3ec [8fae273c]Fabric side Handling ChaincodeMessage of type: INIT in state ready
2018-07-23 11:57:42.424 UTC [chaincode] filterError -> DEBU 3ed Ignoring NoTransitionError: no transition
2018-07-23 11:57:42.424 UTC [chaincode] processStream -> DEBU 3ee [8fae273c]sending state message INIT
2018-07-23 11:57:42.425 UTC [dev-peer1.org1.example.com-mycc-1.0] func2 -> INFO 3ef ex02 Init
2018-07-23 11:57:42.426 UTC [dev-peer1.org1.example.com-mycc-1.0] func2 -> INFO 3f0 Aval = 100, Bval = 200
2018-07-23 11:57:42.426 UTC [chaincode] processStream -> DEBU 3f1 [8fae273c]Received message PUT_STATE from shim
2018-07-23 11:57:42.426 UTC [chaincode] handleMessage -> DEBU 3f2 [8fae273c]Fabric side Handling ChaincodeMessage of type: PUT_STATE in state ready
2018-07-23 11:57:42.426 UTC [chaincode] filterError -> DEBU 3f3 Ignoring NoTransitionError: no transition
2018-07-23 11:57:42.426 UTC [chaincode] func1 -> DEBU 3f4 [8fae273c]state is ready
2018-07-23 11:57:42.426 UTC [chaincode] isValidTxSim -> ERRO 3f5 [[8fae273c PUT_STATE ERROR]]No ledger context for %!!(MISSING)s(MISSING). Sending %!!(MISSING)s(MISSING)
2018-07-23 11:57:42.426 UTC [chaincode] 1 -> DEBU 3f6 [8fae273c]enterBusyState trigger event ERROR
2018-07-23 11:57:42.426 UTC [chaincode] processStream -> DEBU 3f7 [8fae273c]Move state message ERROR
2018-07-23 11:57:42.426 UTC [chaincode] handleMessage -> DEBU 3f8 [8fae273c]Fabric side Handling ChaincodeMessage of type: ERROR in state ready
2018-07-23 11:57:42.426 UTC [chaincode] handleMessage -> DEBU 3f9 [8fae273cee64d4743ca181f848f796a9b2649869bccf5647fa335e20c59ab850]HandleMessage- COMPLETED. Notify
2018-07-23 11:57:42.427 UTC [chaincode] notify -> DEBU 3fa notifier Txid:8fae273cee64d4743ca181f848f796a9b2649869bccf5647fa335e20c59ab850, channelID: does not exist
2018-07-23 11:57:42.427 UTC [chaincode] processStream -> DEBU 3fb [8fae273c]sending state message ERROR
2018-07-23 11:57:42.428 UTC [chaincode] processStream -> DEBU 3fc [8fae273c]Received message ERROR from shim
2018-07-23 11:57:42.428 UTC [chaincode] processStream -> ERRO 3fd Got error: [[8fae273c PUT_STATE ERROR]]No ledger context for %!s(MISSING). Sending %!s(MISSING)
2018-07-23 11:57:42.428 UTC [chaincode] handleMessage -> DEBU 3fe [8fae273c]Fabric side Handling ChaincodeMessage of type: ERROR in state ready
2018-07-23 11:57:42.428 UTC [chaincode] handleMessage -> DEBU 3ff [8fae273cee64d4743ca181f848f796a9b2649869bccf5647fa335e20c59ab850]HandleMessage- COMPLETED. Notify
2018-07-23 11:57:42.428 UTC [chaincode] notify -> DEBU 400 notifier Txid:8fae273cee64d4743ca181f848f796a9b2649869bccf5647fa335e20c59ab850, channelID: does not exist
2018-07-23 11:57:42.428 UTC [dev-peer1.org1.example.com-mycc-1.0] func2 -> INFO 401 2018-07-23 11:57:42.427 UTC [shim] handlePutState -> ERRO 001 [8fae273c]Received ERROR. Payload: [[8fae273c PUT_STATE ERROR]]No ledger context for %!s(MISSING). Sending %!s(MISSING)
2018-07-23 11:57:42.428 UTC [dev-peer1.org1.example.com-mycc-1.0] func2 -> INFO 402 2018-07-23 11:57:42.427 UTC [shim] 2 -> ERRO 002 [[8fae273c ERROR]]Init get error response [%!s(MISSING)]. Sending %!s(MISSING)
2018-07-23 11:58:12.424 UTC [chaincode] Execute -> DEBU 403 Exit
2018-07-23 11:58:12.425 UTC [endorser] callChaincode -> DEBU 404 [mychannel][8fae273c] Exit
2018-07-23 11:58:12.425 UTC [endorser] simulateProposal -> ERRO 405 [mychannel][8fae273c] failed to invoke chaincode name:"lscc" , error: timeout expired while executing transaction
anzalbeg (Mon, 23 Jul 2018 12:11:00 GMT):
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Execute
/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/chaincode_support.go:813
github.com/hyperledger/fabric/core/chaincode.Execute
/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/exectransaction.go:58
github.com/hyperledger/fabric/core/endorser.(*SupportImpl).Execute
/opt/gopath/src/github.com/hyperledger/fabric/core/endorser/support.go:93
github.com/hyperledger/fabric/core/endorser.(*Endorser).callChaincode
/opt/gopath/src/github.com/hyperledger/fabric/core/endorser/endorser.go:173
github.com/hyperledger/fabric/core/endorser.(*Endorser).simulateProposal
/opt/gopath/src/github.com/hyperledger/fabric/core/endorser/endorser.go:287
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposal
/opt/gopath/src/github.com/hyperledger/fabric/core/endorser/endorser.go:513
github.com/hyperledger/fabric/core/handlers/auth/filter.(*expirationCheckFilter).ProcessProposal
/opt/gopath/src/github.com/hyperledger/fabric/core/handlers/auth/filter/expiration.go:61
github.com/hyperledger/fabric/core/handlers/auth/filter.(*filter).ProcessProposal
/opt/gopath/src/github.com/hyperledger/fabric/core/handlers/auth/filter/filter.go:31
github.com/hyperledger/fabric/protos/peer._Endorser_ProcessProposal_Handler
/opt/gopath/src/github.com/hyperledger/fabric/protos/peer/peer.pb.go:112
github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).processUnaryRPC
/opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:781
github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).handleStream
/opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:981
github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1
/opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:551
runtime.goexit
/opt/go/src/runtime/asm_amd64.s:2337
failed to execute transaction
2018-07-23 11:58:12.425 UTC [endorser] simulateProposal -> DEBU 406 [mychannel][8fae273c] Exit
2018-07-23 11:58:12.425 UTC [lockbasedtxmgr] Done -> DEBU 407 Done with transaction simulation / query execution [8fae273cee64d4743ca181f848f796a9b2649869bccf5647fa335e20c59ab850]
2018-07-23 11:58:12.425 UTC [endorser] ProcessProposal -> DEBU 408 Exit: request from%!(EXTRA string=10.128.0.3:60688)
anzalbeg (Mon, 23 Jul 2018 12:11:35 GMT):
i am getting this issue
[e9585782 PUT_STATE ERROR]]No ledger context for %!!(MISSING)s(MISSING). Sending %!!(MISSING)s(MISSING) ERRO 6a2 [bloqchannel][e9585782] failed to invoke chaincode name:"lscc" , error: timeout expired while executing transaction
bobzhao (Mon, 23 Jul 2018 15:26:14 GMT):
Has joined the channel.
iamksseo (Tue, 24 Jul 2018 02:00:21 GMT):
Has joined the channel.
jayeshjawale95 (Tue, 24 Jul 2018 09:17:01 GMT):
Require help with production setup
jayeshjawale95 (Tue, 24 Jul 2018 09:17:01 GMT):
Require help with production setup, kubernetes, fabric, flannel, kafka orderers
jayeshjawale95 (Tue, 24 Jul 2018 09:17:01 GMT):
Require help with production setup, kubernetes, fabric, flannel, kafka orderers, and how chaincode containers will be spawn
fabiomolinar (Tue, 24 Jul 2018 18:12:26 GMT):
Has joined the channel.
jakereps (Tue, 24 Jul 2018 22:46:23 GMT):
Has joined the channel.
MonnyClara (Wed, 25 Jul 2018 07:01:43 GMT):
@maheshwarishikha Maybe you can check if the env variable PEER_LISTEN_ADDRESS is correctly set. (I'm not sure of the variable name)
Muffi (Thu, 26 Jul 2018 06:53:48 GMT):
Has joined the channel.
naviat (Thu, 26 Jul 2018 09:01:42 GMT):
Has joined the channel.
VictorStroganov (Fri, 27 Jul 2018 07:37:57 GMT):
Has joined the channel.
danhawker (Fri, 27 Jul 2018 15:04:56 GMT):
Has joined the channel.
zjubfd (Sat, 28 Jul 2018 10:13:34 GMT):
Has joined the channel.
sudhir.kumawat (Sun, 29 Jul 2018 05:39:18 GMT):
Hi Everyone, Is there any support yet to run chaincode in kubernetes pod rather than docker container in any version of hyperledger. I want to run chaincode in pod for clustering of chaincode to achieve run fabric network in fault tolerance manner??
Or there is any process to run chaincode container in all peers of an organisation running on different machine . Because it run by peer on same machine as peers running in docker container.
antonikonovalov (Tue, 31 Jul 2018 10:44:11 GMT):
Has joined the channel.
kleniu (Tue, 31 Jul 2018 14:13:47 GMT):
Has joined the channel.
Kyroy (Wed, 01 Aug 2018 15:44:04 GMT):
Has joined the channel.
smeyers (Wed, 01 Aug 2018 22:43:46 GMT):
Has joined the channel.
zmaro (Fri, 03 Aug 2018 15:21:16 GMT):
Has joined the channel.
akshay.sood (Sat, 04 Aug 2018 06:47:31 GMT):
Has joined the channel.
kmohanar1 (Tue, 07 Aug 2018 09:34:49 GMT):
Has joined the channel.
huxiangdong (Wed, 08 Aug 2018 00:20:38 GMT):
Has joined the channel.
shahrukh1426 (Wed, 08 Aug 2018 08:34:24 GMT):
Has joined the channel.
mavericklam (Tue, 14 Aug 2018 04:42:25 GMT):
Has joined the channel.
JOYELIN (Tue, 21 Aug 2018 07:16:13 GMT):
Has left the channel.
DannyWong (Tue, 21 Aug 2018 10:15:31 GMT):
https://medium.com/kokster/simpler-setup-for-hyperledger-fabric-on-kubernetes-using-docker-in-docker-8346f70fbe80
bdjidi (Tue, 21 Aug 2018 22:48:26 GMT):
Has joined the channel.
aatkddny (Thu, 23 Aug 2018 20:42:52 GMT):
Has joined the channel.
aatkddny (Thu, 23 Aug 2018 20:43:54 GMT):
I asked this in general - not knowing this channel existed. This might be a better forum for it - although the activity here is pretty limited.
Anyone have experience using kubernetes with docker in a mac? Version 18.06.0-ce-mac70 (26399) using Kubernetes: v1.10.3
I'm having trouble getting it to recognize a PVC with a local NFS.
The NFS is fine - it mounts ok in a terminal as expected. The PV looks good. The PVC looks good.
When I go to deploy though the pod pulls this - ```Unable to mount volumes for pod "blockchain-org1peer1-56ffc95965-8n6nz_default(a131c320-a640-11e8-ba48-025000000001)": timeout expired waiting for volumes to attach or mount for pod "default"/"blockchain-org1peer1-56ffc95965-8n6nz". list of unmounted volumes=[sharedvolume]. list of unattached volumes=[sharedvolume default-token-7z9gr]```
I'm thinking it's a problem with NFS inside ~this~ type of install but kubernetes isn't my chosen specialist subject. Anyone hit this?
Joe-mcgee (Fri, 24 Aug 2018 16:42:28 GMT):
Has joined the channel.
vanclief (Sun, 26 Aug 2018 14:35:17 GMT):
Has joined the channel.
pankajcheema (Mon, 27 Aug 2018 05:40:10 GMT):
Hi All
pankajcheema (Mon, 27 Aug 2018 05:40:29 GMT):
Is there any good tutorial for deploying fabric on Kubernetes
pankajcheema (Mon, 27 Aug 2018 05:40:29 GMT):
?
qiangjiyi (Tue, 28 Aug 2018 02:13:51 GMT):
Has joined the channel.
pankajcheema (Tue, 28 Aug 2018 08:55:57 GMT):
HI Experts
pankajcheema (Tue, 28 Aug 2018 08:56:09 GMT):
I am getting this error in kubernetes
pankajcheema (Tue, 28 Aug 2018 08:56:11 GMT):
```Error: could not assemble transaction, err Proposal response was not successful, error code 500, msg failed to execute transaction 9e48ffcac3df1cee1376df91273e71cf61e4b5c995db79842f56d29e5abbd151: error starting container: error starting container: API error (404): No such network: ${COMPOSE_PROJECT_NAME}_byfn```
pankajcheema (Tue, 28 Aug 2018 08:56:31 GMT):
env for peer ```env:
- name: CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE
value: "${COMPOSE_PROJECT_NAME}_byfn"
- name: CORE_VM_ENDPOINT
value: "unix:///host/var/run/docker.sock"
- name: CORE_PEER_ID
value: "peer0.example.com"```
pankajcheema (Tue, 28 Aug 2018 08:56:42 GMT):
Anyone knows the solution
pankajcheema (Tue, 28 Aug 2018 10:10:05 GMT):
@DannyWong
pankajcheema (Tue, 28 Aug 2018 10:10:24 GMT):
@Luke_Chen
pankajcheema (Tue, 28 Aug 2018 15:09:40 GMT):
https://stackoverflow.com/questions/52059775/hyperledger-fabric-with-kubernetes-not-able-to-instantiate-chaincode
pankajcheema (Tue, 28 Aug 2018 16:45:11 GMT):
Any expert here?
pankajcheema (Tue, 28 Aug 2018 16:48:19 GMT):
@anzalbeg did you find solution to that issue?
Ibraxos (Wed, 29 Aug 2018 04:31:33 GMT):
Has joined the channel.
holzeis (Wed, 29 Aug 2018 07:46:39 GMT):
Has joined the channel.
holzeis (Wed, 29 Aug 2018 08:21:50 GMT):
Hi, I am trying to startup a fabric network in kubernetes and currently stuck that the peer is not able to startup the chaincode server - see below:
peer
```
2018-08-28 14:54:40.406 UTC [nodeCmd] createChaincodeServer -> WARN 088 peer.chaincodeListenAddress is not set, using peer0-service:7052
2018-08-28 14:54:40.409 UTC [nodeCmd] createChaincodeServer -> ERRO 089 Error creating GRPC server: listen tcp 10.108.196.51:7052: bind: cannot assign requested address
2018-08-28 14:54:40.409 UTC [nodeCmd] serve -> CRIT 08a Failed to create chaincode server: listen tcp 10.108.196.51:7052: bind: cannot assign requested address
panic: Failed to create chaincode server: listen tcp 10.108.196.51:7052: bind: cannot assign requested address
```
The host docker sock is mounted, any suggestions on solving that issue?
RobertDiebels (Wed, 29 Aug 2018 14:11:38 GMT):
Has joined the channel.
Luke_Chen (Wed, 29 Aug 2018 16:04:00 GMT):
Hi, We developed a Helm/Charts for deploying Hyperledger Fabric, please check this link for more details http://www.think-foundry.com/hyperledger-fabric-deployment-using-helm-chart/, beside we also consider to add this tool to Cello, take the fully advantage and flexibility of the Helm
kylekim (Thu, 30 Aug 2018 03:20:03 GMT):
Has joined the channel.
SunilHirole (Thu, 30 Aug 2018 09:30:51 GMT):
Has joined the channel.
jdfigure (Thu, 30 Aug 2018 17:42:27 GMT):
Has joined the channel.
iramiller (Thu, 30 Aug 2018 19:35:59 GMT):
Has joined the channel.
iramiller (Thu, 30 Aug 2018 19:36:52 GMT):
@Luke_Chen have you explored configmaps, secrets, and network policies inypur kubernetes environment?
iramiller (Thu, 30 Aug 2018 19:38:02 GMT):
In our kubernetes environment we were able to remove dependencies on NFS which made things much more flexible for GKE deployments
iramiller (Thu, 30 Aug 2018 19:40:17 GMT):
My current effort of modifying the peer and chaincode process to natively support kubernetes for workload scheduling seems like a very promising path for making fabric work well on kubernetes, especially in secure, production, managed cloud environments.
iramiller (Thu, 30 Aug 2018 19:45:22 GMT):
Also @Luke_Chen you may want to look into using stateful sets for peers... the consistency guarantees offered by k8s and the naming/dns were very helpful for me when setting things up with more peer instances.
kirin13 (Fri, 31 Aug 2018 02:56:38 GMT):
Has joined the channel.
Luke_Chen (Fri, 31 Aug 2018 03:33:42 GMT):
@iramiller I have not make that so far
Luke_Chen (Fri, 31 Aug 2018 03:33:42 GMT):
@iramiller I have not made that so far
Luke_Chen (Fri, 31 Aug 2018 03:34:54 GMT):
@iramiller what are stateful sets for peers ? do you have any docs that I can track?
vanitas92 (Fri, 31 Aug 2018 06:37:32 GMT):
Hello everyone! The newly introduced service discovery in latest release has arised some questions about the best approach on how to deploy the latest version on k8s. Should we use the statefulset for deploying the components of an hyperledger fabric network (peers, orderer) and use headless service so the service discovery is handled by the application or should we stick with deployments and non headless services? Has anyone had some experience with that approach? Thank you!
RobertDiebels (Fri, 31 Aug 2018 08:53:46 GMT):
Using a StatefulSet mean you can't use ConfigMaps or Secrets properly since you can only mount them into the entire set instead of one Pod. Same goes for mounting other volumes they're shared across the set afaik.
RobertDiebels (Fri, 31 Aug 2018 08:53:46 GMT):
Using a StatefulSet means you can't use ConfigMaps or Secrets properly since you can only mount them into the entire set instead of one Pod. Same goes for mounting other volumes they're shared across the set afaik.
RobertDiebels (Fri, 31 Aug 2018 08:56:54 GMT):
So if you'd want to mount a certificate into only one Pod that's a no-go.
RobertDiebels (Fri, 31 Aug 2018 08:56:54 GMT):
So if you'd want to mount a single certificate as a ConfigMap/Secret into one Pod only that's a no-go.
iramiller (Fri, 31 Aug 2018 12:14:26 GMT):
In my case the certificates are installed on the peers specific volume and are created when that pod joins the pool. You need a stateful set if you care about pod disruption budgets.
iramiller (Fri, 31 Aug 2018 12:20:11 GMT):
If you using a set of peers with in a single org for redundancy then having them managed as a logical group is useful because the configuration of that group is handled as a set. Therefore the configmaps and secrets are shared across the group and this is a good thing. Peer specific configuration secrets (certificates mentioned above only) are stored on the peer’s individual data volumes.
RobertDiebels (Fri, 31 Aug 2018 12:25:33 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=BijifdNJ4Z4yauFzP) I mentioned it because you stated this.
RobertDiebels (Fri, 31 Aug 2018 12:29:15 GMT):
I actually use a combination of Deployments + ConfigMaps/Secrets to mount my certificates.
iramiller (Fri, 31 Aug 2018 12:29:25 GMT):
Yes, that comment was made because I was interested in getting an understanding of how others use kubernetes. Was his post a summary or not.
iramiller (Fri, 31 Aug 2018 12:31:24 GMT):
There are many ways to apply the tools in kubernetes... it is especially interesting to hear how others are taking a project which is new and unique and applying it to the environment.
RobertDiebels (Fri, 31 Aug 2018 12:33:28 GMT):
Exactly. That's why I thought I'd add to the conversation and provide some info about the Secrets/ConfigMaps + StatefulSet.
iramiller (Fri, 31 Aug 2018 12:34:28 GMT):
:arrow_up:
RobertDiebels (Fri, 31 Aug 2018 12:35:13 GMT):
I take the long route and copy the ConfigMaps into a Peer specific volume since version 1.0.x doesn't support loading PemMaterial from symlinks.
iramiller (Fri, 31 Aug 2018 12:35:17 GMT):
My current focus on removing the direct client docker invocations from the peer platform and chaincode areas is all about making our implementation more kubernetes native. It isn’t as easy to make this software run well in the cloud as I would have liked and the feel I have for it is that the native bare metal install case was taken as the readily approach in design.
RobertDiebels (Fri, 31 Aug 2018 12:36:16 GMT):
Haha I know. I've used a 'hack' to fix the chaincode dns issues without altering the dockerconfig on the host.
RobertDiebels (Fri, 31 Aug 2018 12:36:43 GMT):
There was an issue in Jira somewhere about adding a kubernetes controlle I think :thinking:
RobertDiebels (Fri, 31 Aug 2018 12:36:43 GMT):
There was an issue in Jira somewhere about adding a kubernetes controller I think :thinking:
iramiller (Fri, 31 Aug 2018 12:37:05 GMT):
When I explained the chaincode container process to our CISO... his face was pretty priceless.
RobertDiebels (Fri, 31 Aug 2018 12:37:21 GMT):
Haha I can imagine xD
iramiller (Fri, 31 Aug 2018 12:37:46 GMT):
I don’t want to leave the wrong impression though. This is good work.
RobertDiebels (Fri, 31 Aug 2018 12:37:59 GMT):
Ow yes obviously. But it still has some issues.
iramiller (Fri, 31 Aug 2018 12:38:08 GMT):
Our environement and use cases are just custom
iramiller (Fri, 31 Aug 2018 12:39:40 GMT):
The kubernetes controller integration isn’t hard fwiw. I hooked in next to the couple places were the Docker controller library is and use viper for an enable flag, kubernetes client to self configure from ambient environment.
RobertDiebels (Fri, 31 Aug 2018 12:40:27 GMT):
Did you create a fork to do this?
iramiller (Fri, 31 Aug 2018 12:40:33 GMT):
That does mean that currently I use the default service account in that namespace to schedule pods. But I can add that configuration later.
iramiller (Fri, 31 Aug 2018 12:40:48 GMT):
Yes we have an internal fork for this.
RobertDiebels (Fri, 31 Aug 2018 12:41:11 GMT):
Think you could contribute the kubernetescontroller? Probably a lot of people who would thank you :P
RobertDiebels (Fri, 31 Aug 2018 12:41:19 GMT):
One of which being myself haha
iramiller (Fri, 31 Aug 2018 12:42:08 GMT):
I believe that Corp policy is that we want to do that.
iramiller (Fri, 31 Aug 2018 12:42:43 GMT):
I didn’t replace the Docker controller. Or make the kubernetes one a complete stand alone.
iramiller (Fri, 31 Aug 2018 12:43:24 GMT):
That is too big of a change. There are protobufs hardcoded for Docker as the environment
RobertDiebels (Fri, 31 Aug 2018 12:44:04 GMT):
Ow damn..
iramiller (Fri, 31 Aug 2018 12:44:16 GMT):
I essentially added a kubernetes controller in the model of the Docker one that is there... then modified the existing docker one to detect kubernetes when initializing.
iramiller (Fri, 31 Aug 2018 12:45:26 GMT):
In this way when the existing code paths run they follow the Docker paths like normal. Then my code is switched in during the create vm step as appropriate and we use k8s from that point.
RobertDiebels (Fri, 31 Aug 2018 12:45:29 GMT):
Still anything could be of value I guess. I don't know how far the work on the kubernetescontroller got.
RobertDiebels (Fri, 31 Aug 2018 12:45:45 GMT):
Don't recall the issue I was reading on it either..
iramiller (Fri, 31 Aug 2018 12:46:48 GMT):
The kubernetes controller is self contained and could be a first class citizen like the Docker one. But the changes to support that are bigger than I want on what I expected to be a long running fork for our needs
RobertDiebels (Fri, 31 Aug 2018 12:46:59 GMT):
Found it -> https://jira.hyperledger.org/browse/FAB-7406?jql=text%20~%20%22kubernetes%20controller%22
RobertDiebels (Fri, 31 Aug 2018 12:48:49 GMT):
Yea I can imagine. I had a tough enough time trying to apply the DNS-options so the created docker containers would be able to find the kubernetes network and vice-versa.
RobertDiebels (Fri, 31 Aug 2018 12:49:15 GMT):
Don't want to think about a standalone k8s controller
iramiller (Fri, 31 Aug 2018 12:56:49 GMT):
Ok. Very useful thread. Thanks for that. My comments... the approach described of building containers in the cluster is just not going to fly. At that point somewhere the Docker sock is still mounted and we are talking about admin rights level code. My method uses the base images and deploys a tarball into the chaincode pod using the same builder patterns as described. No chaincode specific images
iramiller (Fri, 31 Aug 2018 12:58:40 GMT):
I’m working through the platform side of this process at the moment and the kubernetes job approach to build the chaincode seems like a good concept so far.
iramiller (Fri, 31 Aug 2018 13:00:19 GMT):
The points in that thread about not adding dependencies to the core project for kubernetes ¯\_(ツ)_/¯ I don’t have a choice because the hooks to add this are not present and the existing code is hard coded to use Docker. A future version with support from the core team could add those hooks and push this into a standalone library.
RobertDiebels (Fri, 31 Aug 2018 13:12:15 GMT):
I don't see why they shouldn't add dependencies for Kubernetes. Depends on how much they really want to enable deploying to Kubernetes.
RobertDiebels (Fri, 31 Aug 2018 13:12:52 GMT):
Afaic it's absolutely preferable to Docker Compose and the like.
iramiller (Fri, 31 Aug 2018 13:18:15 GMT):
Docker compose isn’t for production anything. Docker swarm doesn’t really have a future... kubernetes is the cloud agnostic container orchestration solution in open source today...
iramiller (Fri, 31 Aug 2018 13:19:44 GMT):
It seems like there isn’t as much focus on production systems and system design in this project as I would have expected. Lots of effort into the trial and experiment modes... come to think of it, that really reflects the state of this area in industry perfectly.
RobertDiebels (Fri, 31 Aug 2018 13:20:51 GMT):
Industry as in SE in general or blockchain-related? Either way I agree.
iramiller (Fri, 31 Aug 2018 13:21:06 GMT):
Blockchain specific.
RobertDiebels (Fri, 31 Aug 2018 13:22:03 GMT):
Agree. Projects are still young though. So lots of potential for growth and maturing.
iramiller (Fri, 31 Aug 2018 13:23:03 GMT):
Which is why the test modes are most important honestly. Running something this dynamic long term in production is going to be a serious endeavor for the next couple years
iramiller (Fri, 31 Aug 2018 13:24:50 GMT):
That all said... my helm charts and minikube give me a full fabric that self-configured and is ready to run in about 2-3min on my MacBook... is an awesome example of development when that exact same chart deploys to GKE scaled up.
RobertDiebels (Fri, 31 Aug 2018 13:24:50 GMT):
I figure they'll get it right. Big corps backing the Hyperledger project so they'll make sure it takes off properly.
RobertDiebels (Fri, 31 Aug 2018 13:26:02 GMT):
Well if you find the time and approval to publish please post them here :thumbsup:
iramiller (Fri, 31 Aug 2018 13:26:44 GMT):
I can certainly share what I know regardless of getting patches ready for release.
iramiller (Fri, 31 Aug 2018 13:28:15 GMT):
Next week I will have to see if we can get an official word on what the engagement process needs to look like with regards to the open source contributions. As a developer I feel it needs to happen as that is what makes open source work.
iramiller (Fri, 31 Aug 2018 13:30:34 GMT):
@Luke_Chen’s blog post looks like a fantastic way to share that can pass the communications office before it goes out. That might end up being the way to get started.
Luke_Chen (Fri, 31 Aug 2018 14:12:03 GMT):
@iramiller @RobertDiebels HI, just saw your conversations and got inspired, beside the Kubernetes controller, we are also facing a mapping problem between domain of certificate and service name of Kubernetes
Luke_Chen (Fri, 31 Aug 2018 14:16:50 GMT):
Regularly, every peer or orderer has a domain name, but it's difficult to identify those peer or orderer with their domain name in Kubernetes, do you have any solution with this problem?
Luke_Chen (Fri, 31 Aug 2018 14:22:01 GMT):
beside the original purpose of using NFS server as shared storage is providing a data backup option for peers and orderers, but afterward we use it to store the certificates :joy:
RobertDiebels (Fri, 31 Aug 2018 14:38:14 GMT):
I set the domain of each organization to the namespace it will be in. Kubernetes services then allow DNS mapping to: `
RobertDiebels (Fri, 31 Aug 2018 14:40:08 GMT):
Don't know if that answers your questin. Will need a bit more clarification otherwise :P
RobertDiebels (Fri, 31 Aug 2018 14:40:08 GMT):
Don't know if that answers your question. Will need a bit more clarification otherwise :P
iramiller (Fri, 31 Aug 2018 15:02:39 GMT):
Our peers in statefulsets approach means that the peer will have a consistent hostname of `peer-2` a set name of `peer-n.peer` and a local dns name of `peer-n.peer.namespace.svc.cluster.local` where n is typically `0..2`. I add these names to our peer certificate as alternates when creating/registering with the ca.
iramiller (Fri, 31 Aug 2018 15:05:49 GMT):
The suffix search portion of kubernetes dns can be really frustrating as the number of suffixes is limited to 6 for the hosts and cloud environments like GKE add some along with the typical kubernetes ones. The `namespace.svc.cluster.local` is a default one that will be present so we can always find a specific peer using `peer-n.peer` on any host in the namespace.
am (Fri, 31 Aug 2018 16:16:34 GMT):
Has joined the channel.
vwagner (Fri, 31 Aug 2018 19:34:54 GMT):
Has joined the channel.
aatkddny (Sat, 01 Sep 2018 17:15:04 GMT):
Trying to set up a peer with couchdb in the same pod.
Having a problem using it with a userid and password
Using the same properties in both containers -
```
- name: COUCHDB_USER
value: peer0user
- name: COUCHDB_PASSWORD
value: peer0password
```
aatkddny (Sat, 01 Sep 2018 17:15:04 GMT):
Trying to set up a peer with couchdb in the same pod.
Having a problem using it with a userid and password
Using the same properties in both containers -
```
- name: COUCHDB_USER
value: peer0user
- name: COUCHDB_PASSWORD
value: peer0password
```
When it tries to bring the peer up I get "you are not an admin user".
Same code works fine in base docker - has anyone come across this, and if so what am I missing?
aatkddny (Sat, 01 Sep 2018 17:15:04 GMT):
Trying to set up a peer with couchdb in the same pod.
Having a problem using it with a userid and password
Using the same properties in both containers -
```
- name: COUCHDB_USER
value: peer0user
- name: COUCHDB_PASSWORD
value: peer0password
```
When it tries to bring the peer up I get "you are not an admin user".
Same code works fine in base docker - has anyone come across this, and if so what am I missing?
The whole thing works fine in party mode.
Luke_Chen (Sun, 02 Sep 2018 06:55:16 GMT):
I mentioned about the naming rule of the service's name in Kubernetes, because I don't want to bring such restriction to user, user should be able to create their organizations with any domain name they want, such like org1.example.com, how do this domain map into the Kubernetes?
Luke_Chen (Sun, 02 Sep 2018 06:55:16 GMT):
I mentioned about the naming rule of the service's name in Kubernetes, because I don't want to bring such restriction to user, user should be able to create their organizations with any domain name they want, such like org1.example.com, the problem is how to map this domain into the Kubernetes?
Luke_Chen (Sun, 02 Sep 2018 07:04:09 GMT):
more especially, we required traffic among peers and orderes need to be encrypted by TLS protocol, however the TLS certificate signed for a specified peer is format like "peerN.org1.example.com", for example, in peer0, then it will have TLS cert with DN= "peer0.org1.example.com" and Kubernetes service name "peer0-org1".
Luke_Chen (Sun, 02 Sep 2018 07:04:09 GMT):
more especially, we required traffic among peers and orderes need to be encrypted by TLS protocol, however the TLS certificate signed for a specified peer is format like "peerN.org1.example.com", for example, peer0, then it will have TLS cert with DN= "peer0.org1.example.com" and Kubernetes service name "peer0-org1".
Luke_Chen (Sun, 02 Sep 2018 07:07:13 GMT):
Then problem is, TLS handshake won't pass when you try to connect a service with domain "peer0-org1", which is mismatched with the DN field specified in the TLS certificate owned by the service.
Luke_Chen (Sun, 02 Sep 2018 07:07:13 GMT):
Then problem is, TLS handshake won't pass when you try to connect a service with domain "peer0-org1", the domain is mismatched with the DN field specified in the TLS certificate owned by the service.
Luke_Chen (Sun, 02 Sep 2018 07:12:34 GMT):
And this is the question I am struggling , I really hope you can give me some advises. :joy::
Luke_Chen (Sun, 02 Sep 2018 07:12:34 GMT):
And this is the question I am struggling , I really hope you can give me some advises. :joy:
RobertDiebels (Sun, 02 Sep 2018 07:21:09 GMT):
If they're going to use org1.example.com they probably own the domain, so you could setup a reverse-proxy like NGinx. Expose NGinx through a LoadBalancer service on a specific IP, direct the domain org1.example.com to that IP and finally setup NGinx configuration so that it maps org1.example.com to the namespace `org1` internally.
iramiller (Sun, 02 Sep 2018 18:25:28 GMT):
You can change the names added to the certificates used by the peers pretty easily... I reworked the certificate process because kubernetes uses `name-n` for sets instead of `namen` that fabric uses by default.
anchit (Mon, 03 Sep 2018 10:46:57 GMT):
Has joined the channel.
RobertDiebels (Tue, 04 Sep 2018 10:43:26 GMT):
Yes but DNS-wise you would still be bound by the K8s namespaces etc. I think that Luke meant the domains should be anything the user wants them to be.
aatkddny (Tue, 04 Sep 2018 12:10:51 GMT):
Anyone get DinD chaincode instantiation working? I feel like I'm banging my head against a brick wall here.
I'm trying to do the most simplistic thing possible - load CC onto my peer. I'll excerpt the relevant parts of what I have:
DinD
```
- name: dind
image: docker:dind
securityContext:
privileged: true
volumeMounts:
- mountPath: /var/lib/docker
name: dindmount
```
aatkddny (Tue, 04 Sep 2018 12:10:51 GMT):
Anyone get DinD chaincode instantiation working? I feel like I'm banging my head against a brick wall here.
I'm trying to do the most simplistic thing possible - load CC onto my peer. I'll excerpt the relevant parts of what I have:
DinD
```
- name: dind
image: docker:dind
securityContext:
privileged: true
volumeMounts:
- mountPath: /var/lib/docker
name: dindmount
```
Relevant peer config - pruned:
```
- name: CORE_PEER_ADDRESSAUTODETECT
value: "true"
- name: CORE_PEER_CHAINCODELISTENADDRESS
value: 0.0.0.0:7052
- name: CORE_PEER_EXTERNALENDPOINT
value: "true"
- name: CORE_PEER_TLS_ENABLED
value: "false"
- name: CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE
value: dcn
- name: CORE_VM_ENDPOINT
value: http://localhost:2375
- name: GOPATH
value: /opt/gopath
- name: GODEBUG
value: netdns=go
```
dindmount is persistent
When I run like the above I get a "network dcn not found" error.
When I try removing it I get something different that goes wrong.
The API I'm using (java) reports back there's an X509 error - which seems odd because TLS is off and everything else is playing nice afaict.
aatkddny (Tue, 04 Sep 2018 12:10:51 GMT):
Anyone get DinD chaincode instantiation working? I feel like I'm banging my head against a brick wall here.
I'm trying to do the most simplistic thing possible - load CC onto my peer. I'll excerpt the relevant parts of what I have:
DinD
```
- name: dind
image: docker:dind
securityContext:
privileged: true
volumeMounts:
- mountPath: /var/lib/docker
name: dindmount
```
Relevant peer config - pruned:
```
- name: CORE_PEER_ADDRESSAUTODETECT
value: "true"
- name: CORE_PEER_CHAINCODELISTENADDRESS
value: 0.0.0.0:7052
- name: CORE_PEER_EXTERNALENDPOINT
value: "true"
- name: CORE_PEER_TLS_ENABLED
value: "false"
- name: CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE
value: dcn
- name: CORE_VM_ENDPOINT
value: http://localhost:2375
- name: GOPATH
value: /opt/gopath
- name: GODEBUG
value: netdns=go
```
dindmount is persistent
When I run like the above I get a "network dcn not found" error.
When I try removing it I get something different that goes wrong.
The API I'm using (java) reports back there's an X509 error - which seems odd because TLS is off and everything else is playing nice afaict.
Adding a `CORE_PEER_TLS_SERVERHOSTOVERRIDE` variable - which I read as a suggestion somewhere - was a dismal failure.
There's a dearth of searchable examples so I'm reduced to trial and error. If anyone can point me to what I have misconfigured I'd be grateful.
aatkddny (Tue, 04 Sep 2018 12:10:51 GMT):
Anyone get DinD chaincode install and instantiation working? I feel like I'm banging my head against a brick wall here.
I'm trying to do the most simplistic thing possible - load CC onto my peer. I'll excerpt the relevant parts of what I have:
DinD
```
- name: dind
image: docker:dind
securityContext:
privileged: true
volumeMounts:
- mountPath: /var/lib/docker
name: dindmount
```
Relevant peer config - pruned:
```
- name: CORE_PEER_ADDRESSAUTODETECT
value: "true"
- name: CORE_PEER_CHAINCODELISTENADDRESS
value: 0.0.0.0:7052
- name: CORE_PEER_EXTERNALENDPOINT
value: "true"
- name: CORE_PEER_TLS_ENABLED
value: "false"
- name: CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE
value: dcn
- name: CORE_VM_ENDPOINT
value: http://localhost:2375
- name: GOPATH
value: /opt/gopath
- name: GODEBUG
value: netdns=go
```
dindmount is persistent
When I run like the above I get a "network dcn not found" error.
When I try removing it I get something different that goes wrong.
The API I'm using (java) reports back there's an X509 error - which seems odd because TLS is off and everything else is playing nice afaict.
Adding a `CORE_PEER_TLS_SERVERHOSTOVERRIDE` variable - which I read as a suggestion somewhere - was a dismal failure.
There's a dearth of searchable examples so I'm reduced to trial and error. If anyone can point me to what I have misconfigured I'd be grateful.
aatkddny (Tue, 04 Sep 2018 12:10:51 GMT):
Anyone get DinD chaincode install and instantiation working? I feel like I'm banging my head against a brick wall here.
I'm trying to do the most simplistic thing possible - load CC onto my peer. I'll excerpt the relevant parts of what I have:
DinD
```
- name: dind
image: docker:dind
securityContext:
privileged: true
volumeMounts:
- mountPath: /var/lib/docker
name: dindmount
```
Relevant peer config - pruned:
```
- name: CORE_PEER_ADDRESSAUTODETECT
value: "true"
- name: CORE_PEER_CHAINCODELISTENADDRESS
value: 0.0.0.0:7052
- name: CORE_PEER_EXTERNALENDPOINT
value: "true"
- name: CORE_PEER_TLS_ENABLED
value: "false"
- name: CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE
value: dcn
- name: CORE_VM_ENDPOINT
value: http://localhost:2375
- name: GOPATH
value: /opt/gopath
- name: GODEBUG
value: netdns=go
```
dindmount is persistent
When I run like the above I get a "network dcn not found" error.
When I try removing `- name: CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE
value: dcn
` I get something different that goes wrong.
The API I'm using (java) reports back there's an X509 error - which seems odd because TLS is off and everything else is playing nice afaict.
Adding a `CORE_PEER_TLS_SERVERHOSTOVERRIDE` variable - which I read as a suggestion somewhere - was a dismal failure.
There's a dearth of searchable examples so I'm reduced to trial and error. If anyone can point me to what I have misconfigured I'd be grateful.
aatkddny (Tue, 04 Sep 2018 12:10:51 GMT):
Anyone get DinD chaincode install and instantiation working? I feel like I'm banging my head against a brick wall here.
I'm trying to do the most simplistic thing possible - load CC onto my peer. I'll excerpt the relevant parts of what I have:
DinD
```
- name: dind
image: docker:dind
securityContext:
privileged: true
volumeMounts:
- mountPath: /var/lib/docker
name: dindmount
```
Relevant peer config - pruned:
```
- name: CORE_PEER_ADDRESSAUTODETECT
value: "true"
- name: CORE_PEER_CHAINCODELISTENADDRESS
value: 0.0.0.0:7052
- name: CORE_PEER_EXTERNALENDPOINT
value: "true"
- name: CORE_PEER_TLS_ENABLED
value: "false"
- name: CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE
value: dcn
- name: CORE_VM_ENDPOINT
value: http://localhost:2375
- name: GOPATH
value: /opt/gopath
- name: GODEBUG
value: netdns=go
```
dindmount is persistent
When I run like the above I get a "network dcn not found" error.
When I try removing the `CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE` variable I get something different that goes wrong.
The API I'm using (java) reports back there's an X509 error - which seems odd because TLS is off and everything else is playing nice afaict.
Adding a `CORE_PEER_TLS_SERVERHOSTOVERRIDE` variable - which I read as a suggestion somewhere - was a dismal failure.
There's a dearth of searchable examples so I'm reduced to trial and error. If anyone can point me to what I have misconfigured I'd be grateful.
theathibm (Tue, 04 Sep 2018 15:51:56 GMT):
Has joined the channel.
theathibm (Tue, 04 Sep 2018 17:32:00 GMT):
I'm looking for guidance for choosing Docker Swarm .vs. Kubernetes. What is the current guidance, and what information is available for either direction?
iramiller (Tue, 04 Sep 2018 23:05:07 GMT):
The choice between those two would be more about operational considerations than software ones in my opinion.
aatkddny (Wed, 05 Sep 2018 01:06:30 GMT):
Well there are a lot more examples for swarm... OTOH k8s is probably the defacto standard for container management.
theathibm (Wed, 05 Sep 2018 15:52:30 GMT):
does anyone have thoughts about using "cello" ?
smeyers (Wed, 05 Sep 2018 16:55:11 GMT):
anchovim kns
RobertDiebels (Wed, 05 Sep 2018 17:29:22 GMT):
Afaic cello is much to complicated for what it's trying to do + it doesn't work on Windows.
RobertDiebels (Wed, 05 Sep 2018 17:31:26 GMT):
Especially the last part was important for me.
tylerwince (Wed, 05 Sep 2018 20:41:15 GMT):
dind
aatkddny (Thu, 06 Sep 2018 01:56:16 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=LrcfNikbftGRvb3v3) Anyone have a way to synchronize between mysql and fabric-ca so that i can run them in the same pod?
aatkddny (Thu, 06 Sep 2018 01:56:16 GMT):
Anyone have a way to synchronize between mysql and fabric-ca so that i can run them in the same pod?
aatkddny (Thu, 06 Sep 2018 01:56:16 GMT):
Anyone have a way to synchronize between mysql and fabric-ca so that i can run them in the same pod?
aatkddny (Thu, 06 Sep 2018 01:57:15 GMT):
anyone have a way to synchronize between mysql and fabric-ca so i can run them in the same pod that they'd be willing to share?
RobertDiebels (Thu, 06 Sep 2018 07:32:12 GMT):
@holzeis I don't know if you resolved your issue. I seem to have missed your question. Here's a possible solution though. If you're using 1.0.x using `peer0-service:7052` should work. If you're using 1.1 or 1.2 you need to set the `peer.chaincodeListenAddress` to `0.0.0.0:7052`. Since the dns-address `peer0-service` is being resolved to `10.108.196.51` which is the IP address of the service. Not the local-IP of the peer-container. You can check the ClusterIP of the service: `peer0-service` to verify that's what's happening. It's either that or the ClusterIP of the Pod the container is running in.
RobertDiebels (Thu, 06 Sep 2018 07:34:45 GMT):
Also, check that your DNS-address is also using the namespace the service is in. So if the service is named: `peer0-service` and the namespace it is in is named `namespace` the dns-address should be: `peer0-service.namespace:7052`
RobertDiebels (Thu, 06 Sep 2018 07:34:45 GMT):
If you're using 1.0.x check that your DNS-address is also using the namespace the service is in. So if the service is named: `peer0-service` and the namespace it is in is named `namespace` the dns-address should be: `peer0-service.namespace:7052`
RobertDiebels (Thu, 06 Sep 2018 07:35:42 GMT):
It seems to be resolving an IP-address though so I guess it's at least doing something :P
aatkddny (Thu, 06 Sep 2018 16:57:20 GMT):
anyone get a fabric up with TLS?
aatkddny (Thu, 06 Sep 2018 16:57:20 GMT):
anyone get a fabric up on k8s with TLS?
iramiller (Fri, 07 Sep 2018 20:33:50 GMT):
@aatkddny yes we have a fabric running in kubernetes with TLS, it is certainly possible.
yacovm (Fri, 07 Sep 2018 20:42:49 GMT):
@G-DazWilkin ^
G-DazWilkin (Fri, 07 Sep 2018 20:42:50 GMT):
Has joined the channel.
aatkddny (Fri, 07 Sep 2018 22:16:18 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=6pc7f3makkh9rMD7R) so would you care to give me a hint as to the trick? request certs from kubernetes and use those? security isn't my chosen specialist subject - I can get it running using the generated certs in docker, but i'm a bit out of my depth with kubernetes. i've managed to get it to run with dind and no security, but as soon as I try turning tls on it all goes pear shaped.
aatkddny (Fri, 07 Sep 2018 22:16:18 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=6pc7f3makkh9rMD7R) so would you care to give me a hint as to the trick? Request certs from kubernetes and use those in place of the generated TLS ones? Security isn't my chosen specialist subject - I can get it running using the generated certs in docker, but i'm a bit out of my depth with kubernetes.
I've managed to get it to run with dind and no security, but as soon as I try turning TLS on it all goes pear shaped.
kjroger94 (Sun, 09 Sep 2018 12:19:01 GMT):
Has joined the channel.
kjroger94 (Sun, 09 Sep 2018 16:23:42 GMT):
i just started a peer on kubernetes and i saw this log ```
`018-09-09 16:21:15.295 UTC [discovery] NewService -> INFO 10b Created with config TLS: false, authCacheMaxSize: 1000, authCachePurgeRatio: 0.750000
2018-09-09 16:21:15.295 UTC [nodeCmd] registerDiscoveryService -> INFO 10c Discovery service activated
2018-09-09 16:21:15.296 UTC [nodeCmd] serve -> INFO 10d Starting peer with ID=[name:"liqvis" ], network ID=[nid1], address=[10.244.0.61:7051]
2018-09-09 16:21:15.296 UTC [nodeCmd] serve -> INFO 10e Started peer with ID=[name:"liqvis" ], network ID=[nid1], address=[10.244.0.61:7051]
2018-09-09 16:21:15.296 UTC [nodeCmd] func9 -> INFO 10f Starting profiling server with listenAddress = 0.0.0.0:6060`
```
kjroger94 (Sun, 09 Sep 2018 16:24:16 GMT):
why does it say `listenAddress = 0.0.0.0:6060`
kjroger94 (Sun, 09 Sep 2018 16:27:56 GMT):
my env in the deployment yaml is ```
` - name: CORE_PEER_ADDRESS
value: liqvis:7051
- name: CORE_PEER_LISTENADDRESS
value: 0.0.0.0:7051
- name: CORE_PEER_EVENTS_ADDRESS
value: 0.0.0.0:7052
- name: CORE_PEER_GOSSIP_BOOTSTRAP
value: liqvis:7051`
```
yacovm (Sun, 09 Sep 2018 16:28:27 GMT):
there is another service... for profiling, the `pprof` one
yacovm (Sun, 09 Sep 2018 16:28:32 GMT):
you may turn it off
yacovm (Sun, 09 Sep 2018 16:29:17 GMT):
https://github.com/hyperledger/fabric/blob/release-1.2/sampleconfig/core.yaml#L377
kjroger94 (Mon, 10 Sep 2018 01:50:29 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=kejagWjMCy3qgnZf7) @yacovm ok will try, thanks
kjroger94 (Mon, 10 Sep 2018 03:24:34 GMT):
i went through the the core.yaml that is there in my network that has been running for quite sometime. I don't see the core.yaml config being used.
kjroger94 (Mon, 10 Sep 2018 07:13:30 GMT):
how do i run chaincode on kubernetes, too many ways, too confusing
kjroger94 (Mon, 10 Sep 2018 07:13:40 GMT):
i am using aks for k8s
kjroger94 (Mon, 10 Sep 2018 10:20:17 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=Kt55g9CmbmYfyg69w) @greg.haskins can someone give me an example of what a value of this env would look like?
iramiller (Mon, 10 Sep 2018 16:18:12 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=WznH3qrBEoayDcPfS) @aatkddny @aatkddny I need a little bit more specifics on which piece you are having trouble with ... in my environments I have setup my kubernetes deployments to use stateful sets for consistent naming ... this was helpful because pods for peers have consistent naming that is resolvable via kubedns without needing to use headless services. In this type of configuration I can register peers like so: ``` fabric-ca-client enroll -d --enrollment.profile tls -u $ENROLLMENT_URL -M /tmp/tls --csr.hosts "$HOST_NAME,$HOST_NAME.peer,$HOST_NAME.peer.$CORE_PEER_DOMAIN"``` where examples of these variables are `HOST_NAME` as `peer-0`, `CORE_PEER_DOMAIN` is `namespace.svc.cluster.local` giving an overall peer name of `peer-0.peer.namespace.svc.cluster.local`
aatkddny (Mon, 10 Sep 2018 18:22:49 GMT):
I use local storage - I'm on a mac and NFS isn't working.
I'm having trouble getting it to accept the certs I gen with cryptogen when I call a peer with a client. I get a 509 exception. It's fine inside regular docker with same.
iramiller (Mon, 10 Sep 2018 18:58:16 GMT):
We do not use NFS in our K8S cluster. It is possible to setup a fabric without it but if you are basing your system off of the demo scripts then you will need to do some work to get past that step. We use a configmap with the genesis.block, configtx.yaml, channel.tx, and anchors.tx files in it and mount those into pods
iramiller (Mon, 10 Sep 2018 18:59:09 GMT):
Additionally we wait on ports vs waiting on log files for the steps that require that sync...
iramiller (Mon, 10 Sep 2018 18:59:17 GMT):
What is your X509 error?
yacovm (Mon, 10 Sep 2018 20:53:26 GMT):
Is anyone here using Helm charts and has one that works for fabric deployment?
iramiller (Mon, 10 Sep 2018 21:10:31 GMT):
@yacovm we have a custom chart that works for deployment ...
yacovm (Mon, 10 Sep 2018 21:15:30 GMT):
Thanks @iramiller :)
@G-DazWilkin - you were looking for such a chart... ^
yacovm (Mon, 10 Sep 2018 21:15:30 GMT):
Thanks @iramiller :)
@G-DazWilkin - you were looking for such a chart... ^
yacovm (Mon, 10 Sep 2018 21:16:21 GMT):
@iramiller - is it secret?
yacovm (Mon, 10 Sep 2018 21:16:43 GMT):
any chance you'll be willing to help someone that is looking for such a thing? (not me - I tagged him)
iramiller (Mon, 10 Sep 2018 21:16:48 GMT):
No it is not a secret ... but I would need to share it out in pieces :-/ because it depends on some of our own custom containers and start scripts
yacovm (Mon, 10 Sep 2018 21:16:56 GMT):
understood
iramiller (Mon, 10 Sep 2018 21:17:41 GMT):
but basic and important concepts like CAs and orderer/peers are all pretty useable with the default example ... and I can give pointers on the bash changes we made
yacovm (Mon, 10 Sep 2018 21:17:47 GMT):
So, there are 2 hacks which he had to do - wondering if you can advise on the 2 hacks in:
https://medium.com/google-cloud/helm-chart-for-fabric-for-kubernetes-80408b9a3fb6
iramiller (Mon, 10 Sep 2018 21:18:22 GMT):
We are deployed in GKE as well
yacovm (Mon, 10 Sep 2018 21:18:22 GMT):
you can write a response below if you feel generous and kind ;)
yacovm (Mon, 10 Sep 2018 21:18:31 GMT):
(in the medium post)
iramiller (Mon, 10 Sep 2018 21:19:10 GMT):
Yes I was able to overcome those two DNS issues
yacovm (Mon, 10 Sep 2018 21:20:00 GMT):
can you pretty please write in the medium blog post how you solved them? :pray:
yacovm (Mon, 10 Sep 2018 21:20:16 GMT):
you can "leave a response" in the bottom of the page
iramiller (Mon, 10 Sep 2018 21:22:00 GMT):
I will see what I can do... I need to disentangle a couple different efforts to make sure I have it explained correctly. My current system is using chain code in scheduled pods (which I have mentioned in channels here in a few places) and that changes things yet again because we do not need to worry about scheduling containers in the host vm through mounted docker.sock in the peer...
iramiller (Mon, 10 Sep 2018 21:22:49 GMT):
Prior to making the kubernetes controller though we were running with the normal docker.sock schedule pods and we had to deal with this exact DNS issue...
yacovm (Mon, 10 Sep 2018 21:23:13 GMT):
hmm I see
iramiller (Mon, 10 Sep 2018 21:24:42 GMT):
The important change was to use statefulsets for peers so the hostnames matched up using the `peer-1.peer` (where `peer` is the service name for the stateful set) so that DNS could resolve the names
iramiller (Mon, 10 Sep 2018 21:26:37 GMT):
The peer docker vm needs to point to the KUBE_DNS service IP address as well so it can resolve names in kubernetes
```
- name: CORE_VM_DOCKER_HOSTCONFIG_DNS
value: {{ .Values.fabricPeer.hostDNS }}
```
yacovm (Mon, 10 Sep 2018 21:27:08 GMT):
OK, understood... I'll forward your tips to him, many thanks again!
iramiller (Mon, 10 Sep 2018 21:27:32 GMT):
Sure thing. I will see what I can do for a better writeup to attach to his post as well
aatkddny (Tue, 11 Sep 2018 00:11:04 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=u8euvniwyyQwCQqiD) @iramiller Perhaps I didn't explain as well as I could have. We have a different implementation from the samples. The simple test version of it calls cryptogen and makes multiple configtxgen to generate the needful files for security and for the channels. It then generates kubernetes configuration yaml for the four defined organizations. It's set to use DinD for chaincode instantiation, which seems to be working.
All this works fine until I turn on TLS. At that point I start getting certificate exceptions, and I'm not sure what's causing them. External communication to the network is through nodeports to localhost and the calling application uses the java SDK with the peer cert coming from `peerOrganizations/{DOMAIN}/peers/{NAME}/tls/server.crt`.
The same network and the same calling application works just fine in docker using compose, so I'm guessing it's something to do with how kubernetes resolves things but this is a I don't know what I don't know situation. I'm not even sure I'm asking the right questions.
aatkddny (Tue, 11 Sep 2018 00:11:04 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=u8euvniwyyQwCQqiD) @iramiller Perhaps I didn't explain as well as I could have. We have a different implementation from the samples. The simple test version of it calls cryptogen and makes multiple configtxgen to generate the needful files for security and for the channels. It then generates kubernetes configuration yaml for the four defined organizations. It's set to use DinD for chaincode instantiation, which seems to be working.
All this works fine until I turn on TLS. At that point I start getting certificate exceptions, and I'm not sure what's causing them. External communication to the network is through nodeports to localhost and the calling application uses the java SDK with the peer cert coming from `peerOrganizations/{DOMAIN}/peers/{NAME}/tls/server.crt`.
The same network and the same calling application works just fine in docker using compose, so I'm guessing it's something to do with how kubernetes resolves things but this is a I don't know what I don't know situation. I'm not even sure I'm asking the right questions.
aatkddny (Tue, 11 Sep 2018 00:11:04 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=u8euvniwyyQwCQqiD) @iramiller Perhaps I didn't explain as well as I could have. We have a different implementation from the samples. The simple test version of it calls cryptogen and makes multiple configtxgen to generate the needful files for security and for the channels. It then generates kubernetes configuration yaml for the four defined organizations. It's set to use DinD for chaincode instantiation, which seems to be working.
All this works fine until I turn on TLS. At that point I start getting certificate exceptions, and I'm not sure what's causing them.
External communication to the network is through nodeports to localhost. The calling application uses the java SDK with the peer cert coming from `peerOrganizations/{DOMAIN}/peers/{NAME}/tls/server.crt`.
Setting a hostname override doesn't seem to affect things.
The same network and the same calling application works just fine in docker using compose, so I'm guessing it's something to do with how kubernetes resolves things but this is a I don't know what I don't know situation. I'm not even sure I'm asking the right questions.
lotty02cho (Tue, 11 Sep 2018 00:38:08 GMT):
Has joined the channel.
lotty02cho (Tue, 11 Sep 2018 00:52:04 GMT):
How do I start Fabric based on Kubernetes? Doe anyone knows about it? Which reference is good for beginner?
RobertDiebels (Tue, 11 Sep 2018 08:10:23 GMT):
@lotty02cho Check out @Luke_Chen 's recent post wich contains some helm charts to boot a 1.0.x Fabric cluster. [ http://www.think-foundry.com/hyperledger-fabric-deployment-using-helm-chart/ ]
RobertDiebels (Tue, 11 Sep 2018 08:10:31 GMT):
Should be easy enough.
RobertDiebels (Tue, 11 Sep 2018 08:11:15 GMT):
Otheriwise check out a tool I made to do something similar. I'd go with the helm charts though. https://www.npmjs.com/package/kubechain
RobertDiebels (Tue, 11 Sep 2018 08:16:07 GMT):
@yacovm I solved those DNS issues by passing the kubedns IP-address and the DNS config into the container as EnvVar's. You'd need to either access the host or change Fabric's code if you want to fix those issues otherwise.
RobertDiebels (Tue, 11 Sep 2018 08:16:40 GMT):
```
{
"name": "CORE_VM_DOCKER_HOSTCONFIG_DNS",
"value": "
RobertDiebels (Tue, 11 Sep 2018 08:19:41 GMT):
This way I can avoid changing fabric's code and I don't have to alter the dockerconfig file on the host machine. Downside is that the chaincode runs outside of k8s. For my purposes this is good enough, though I can imagine if you want to utilize k8s's full potential running chaincode outside of k8s just won't cut it.
yacovm (Tue, 11 Sep 2018 08:25:21 GMT):
cool, thanks
RobertDiebels (Tue, 11 Sep 2018 08:29:47 GMT):
Won't solve the domain-naming issue for orderer.example.com though. I would have taken a similar approach on that one.
RobertDiebels (Tue, 11 Sep 2018 08:39:09 GMT):
Has left the channel.
cbf (Tue, 11 Sep 2018 15:06:09 GMT):
@iramiller I see in your response to @DazWilkin medium post that you have a forked version of fabric with k8s controller code replacing the docker controller code. Any chance you might contribute that upstream as a configurable alternative?
DazWilkin (Tue, 11 Sep 2018 15:06:10 GMT):
Has joined the channel.
iramiller (Tue, 11 Sep 2018 15:20:28 GMT):
@cbf there are a few hoops to go through before we could release the controller... foremost is that our implementation doesn't support all of the configuration options that are expected for a deployment. We rely heavily on a set of defaults that match our own internal requirements. I am happy to pull out examples of where we integrated in and how we solved scheduling of workloads for others that are able to build and integrate the source code. Until we have a more robust solution releasing the whole thing would likely result in far more support questions than we would be able to field
cbf (Tue, 11 Sep 2018 15:21:39 GMT):
@iramiller thanks, appreciate your circumstance
cbf (Tue, 11 Sep 2018 15:22:04 GMT):
just trying to figure out how we bootstrap and accelerate getting proper k8s support
yacovm (Tue, 11 Sep 2018 15:24:17 GMT):
IMO @cbf - we... should be OK with them releasing what they can, and we (the community should simply fill in the missing gaps)
yacovm (Tue, 11 Sep 2018 15:24:17 GMT):
IMO @cbf - we... should be OK with them releasing what they can, and we (the community) should simply fill in the missing gaps
iramiller (Tue, 11 Sep 2018 15:26:22 GMT):
I feel like our approach is a good starting point ... we essentially integrated directly with the existing docker containers with the exception that we do not use the docker client to schedule workloads... this means that all of the container configuration stuff should map over to kubernetes eventually without change protobufs for a new type
```
// from github.com/hyperledger/fabric/core/container/dockercontroller/dockercontroller.go @ line 100
// NewVM creates a new DockerVM instance
func (p *Provider) NewVM() container.VM {
// At this point check to see if we are in kubernetes
if !kubernetescontroller.InCluster() {
return NewDockerVM(p.PeerID, p.NetworkID)
}
// In a cluster so replace the docker connection with a kubernetes one.
dockerLogger.Info("Kubernetes environment detected. Using K8s API.")
return kubernetescontroller.NewKubernetesAPI(p.PeerID, p.NetworkID)
}
```
iramiller (Tue, 11 Sep 2018 15:30:11 GMT):
There is a good discussion here: https://jira.hyperledger.org/browse/FAB-7406
iramiller (Tue, 11 Sep 2018 15:32:38 GMT):
As part of that I see that the HLF team was not supportive of pulling in the k8s.io client apis as another compile time reference for the peer code with an expressed preference for an external library instead. That approach would not work with the implementation I have above that ties into the docker controller directly to use the in cluster configuration and replace the docker client when the kubernetes environment is detected (and the configuration flag enables the K8s controller)
ColeBoudreau (Tue, 11 Sep 2018 17:18:46 GMT):
Has joined the channel.
ColeBoudreau (Tue, 11 Sep 2018 17:20:04 GMT):
Is it possible to use kubernetes to allow multiple peers (containers) to share chaincode/ledger dbs, in order to conserve memory use?
G-DazWilkin (Tue, 11 Sep 2018 18:00:22 GMT):
@iramiller @yacovm this is *very* interesting. Thank you both for the follow-up. @iramiller your insights in response to my blog post are very helpful too. I'm headed out on vacation today back and the closing on a house so I'm going to be very offline for a while. However, I'm (re)committed to getting this working. How best to proceed? Is there a project space where we can progress this in the open? Or, should we corral, define a path forward and then return here? @iramiller, I'm my handle here without the "G-" prefix at google.com.
kirin (Wed, 12 Sep 2018 03:57:31 GMT):
Has joined the channel.
kirin (Wed, 12 Sep 2018 04:22:41 GMT):
I'm use kubernetes deployment fabric
Basic environmental instructions
operating system： Ubuntu 16.04.3
kubernetes version: v1.11.2
kubernetes network plugin: flannel
kubernetes DNS use: core dns
kubernetes one master and one work node.
fabric use release 1.1.0 docker images.
fabric topology
consensus： kafka
three zookeeper node and four kafka node .
three orderer node and four peer node
The deployment of orderer，
the orderer container log
kirin (Wed, 12 Sep 2018 04:24:10 GMT):
Clipboard - 2018年9月12日中午12点24分
kirin (Wed, 12 Sep 2018 04:24:12 GMT):
I'm use kubernetes deployment fabric
Basic environmental instructions
operating system： Ubuntu 16.04.3
kubernetes version: v1.11.2
kubernetes network plugin: flannel
kubernetes DNS use: core dns
kubernetes one master and one work node.
fabric use release 1.1.0 docker images.
fabric topology
consensus： kafka
three zookeeper node and four kafka node .
three orderer node and four peer node
The deployment of orderer，
the orderer container log
kirin (Wed, 12 Sep 2018 04:24:51 GMT):
Clipboard - 2018年9月12日中午12点24分
underbell (Thu, 13 Sep 2018 06:08:14 GMT):
Has joined the channel.
JaydipMakadia (Thu, 13 Sep 2018 13:11:19 GMT):
Has joined the channel.
aatkddny (Thu, 13 Sep 2018 13:51:25 GMT):
@iramiller I've not been able to look at my problem for a couple of days. I really did a poor job of explaining it, and I conflated two issues I had.
I'm using yaml config scripts I autogenerate and the client runs under the java sdk for full disclosure.
K8S works fine using grpc:// until I turn on TLS and grpcs://.
It works fine in a base docker setup with and without TLS. This is single node on a mac.
The first thing I need to do to run my network is to create a channel. To do this the client needs to contact the orderer. That's exposed using a NodePort service. My call fails to reach the orderer.
The java client gives me this:
```
org.hyperledger.fabric.sdk.exception.TransactionException: Channel xxx-block-channel, send transactions failed on orderer OrdererClient-xxx-block-channel-orderer0-orderer(grpcs://localhost:30020). Reason: timeout after 10000 ms.
at org.hyperledger.fabric.sdk.OrdererClient.sendTransaction(OrdererClient.java:210) ~[fabric-sdk-java-1.2.0.jar:na]
at org.hyperledger.fabric.sdk.Orderer.sendTransaction(Orderer.java:158) [fabric-sdk-java-1.2.0.jar:na]
at org.hyperledger.fabric.sdk.Channel.sendUpdateChannel(Channel.java:509) [fabric-sdk-java-1.2.0.jar:na]
at org.hyperledger.fabric.sdk.Channel.
aatkddny (Thu, 13 Sep 2018 13:51:25 GMT):
@iramiller I've not been able to look at my problem for a couple of days. I really did a poor job of explaining it, and I conflated two issues I had.
I'm using yaml config scripts I autogenerate and the client runs under the java sdk for full disclosure.
K8S works fine using grpc:// until I turn on TLS and grpcs://.
It works fine in a base docker setup with and without TLS. This is single node on a mac.
The first thing I need to do to run my network is to create a channel. To do this the client needs to contact the orderer. That's exposed using a NodePort service. My call fails to reach the orderer.
The java client gives me this:
```
org.hyperledger.fabric.sdk.exception.TransactionException: Channel xxx-block-channel, send transactions failed on orderer OrdererClient-xxx-block-channel-orderer0-orderer(grpcs://localhost:30020). Reason: timeout after 10000 ms.
at org.hyperledger.fabric.sdk.OrdererClient.sendTransaction(OrdererClient.java:210) ~[fabric-sdk-java-1.2.0.jar:na]
at org.hyperledger.fabric.sdk.Orderer.sendTransaction(Orderer.java:158) [fabric-sdk-java-1.2.0.jar:na]
at org.hyperledger.fabric.sdk.Channel.sendUpdateChannel(Channel.java:509) [fabric-sdk-java-1.2.0.jar:na]
at org.hyperledger.fabric.sdk.Channel.
aatkddny (Thu, 13 Sep 2018 13:51:25 GMT):
@iramiller I've not been able to look at my problem for a couple of days. I really did a poor job of explaining it, and I conflated two issues I had.
I'm using yaml config scripts I autogenerate and the client runs under the java sdk for full disclosure.
K8S works fine using grpc:// until I turn on TLS and grpcs://.
It works fine in a base docker setup with and without TLS. This is single node on a mac.
The first thing I need to do to run my network is to create a channel. To do this the client needs to contact the orderer. That's exposed using a NodePort service. My call fails to reach the orderer.
The java client gives me this:
```
org.hyperledger.fabric.sdk.exception.TransactionException: Channel xxx-block-channel, send transactions failed on orderer OrdererClient-xxx-block-channel-orderer0-orderer(grpcs://localhost:30020). Reason: timeout after 10000 ms.
at org.hyperledger.fabric.sdk.OrdererClient.sendTransaction(OrdererClient.java:210) ~[fabric-sdk-java-1.2.0.jar:na]
at org.hyperledger.fabric.sdk.Orderer.sendTransaction(Orderer.java:158) [fabric-sdk-java-1.2.0.jar:na]
at org.hyperledger.fabric.sdk.Channel.sendUpdateChannel(Channel.java:509) [fabric-sdk-java-1.2.0.jar:na]
at org.hyperledger.fabric.sdk.Channel.
iramiller (Thu, 13 Sep 2018 16:22:16 GMT):
I do not recommend using NodePorts for your services. I recommend using a headless service bound directly to your orderer (or statefulsets) so you can have a consistent DNS name and port in your configuration that is easier to reason about. Depending on how you deploy your orderer workloads you can use the DNS names of stateful sets or make a service with specific selectors for each order if you are using a deployment with each specific orderer instance. When diagnosing DNS issues I like to run an instance of the CLI [hyperledger/fabric-ca-tools] and install dnsutils `apt update;apt install -qq -y dnsutils` to check my connections and resolution.
```
root@cli-0:/# nslookup orderer-0
Server: 172.16.0.10
Address: 172.16.0.10#53
** server can't find orderer-0: NXDOMAIN
root@cli-0:/# nslookup orderer-0.orderer
Server: 172.16.0.10
Address: 172.16.0.10#53
Name: orderer-0.orderer.fabric.svc.cluster.local
Address: 10.0.6.64
root@cli-0:/# nslookup orderer-0.orderer.fabric.svc.cluster.local
Server: 172.16.0.10
Address: 172.16.0.10#53
Name: orderer-0.orderer.fabric.svc.cluster.local
Address: 10.0.6.64
##### And DNS lookup of a service
root@cli-0:/# nslookup orderer
Server: 172.16.0.10
Address: 172.16.0.10#53
Name: orderer.fabric.svc.cluster.local
Address: 10.0.6.64```
aatkddny (Thu, 13 Sep 2018 17:26:27 GMT):
I do have headless services for all of the internal inter-pod comms - it's only the ones that I need to hit from my client externally that have NodePorts.
It was my first go around and this was the first time I had to deal with Kubernetes :( I'll add an ingress when I figure out how it works.
Although if I understood all that documentation for it correctly it seems that inter-pod TLS isn't supported without adding something like a reverse proxy.
aatkddny (Thu, 13 Sep 2018 17:26:27 GMT):
I do have headless services for all of the internal inter-pod comms - it's only the ones that I need to hit from my client externally that have NodePorts.
It was my first go around and this was the first time I had to deal with Kubernetes :( I'll add an ingress when I figure out how it works.
Although if I understood all that documentation for it correctly it seems that inter-pod TLS isn't supported without adding something like a reverse proxy to each of the pods. And I don't want to go down that particular rabbit hole.
iramiller (Thu, 13 Sep 2018 17:56:27 GMT):
TCP and UDP services are definitely a weak point in public cloud kubernetes deployments. https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/exposing-tcp-udp-services.md is a starting point ... good luck
aatkddny (Thu, 13 Sep 2018 18:01:33 GMT):
Thanks. I was hoping for a magic bullet but I may just punt and say VPN FTW for now until I have more time to revisit it. This instance is in our datacenter so I can get away with that for a little while.
eenagy (Fri, 14 Sep 2018 02:56:29 GMT):
Has joined the channel.
silliman (Fri, 14 Sep 2018 13:27:55 GMT):
Has joined the channel.
fsl (Mon, 17 Sep 2018 02:23:55 GMT):
Has joined the channel.
luckydogchina (Wed, 19 Sep 2018 10:19:58 GMT):
I find that the host names in the grpc -tls cert of peers is usually include "." , for example, "peer1.org@example.com"
luckydogchina (Wed, 19 Sep 2018 10:19:58 GMT):
I find that host names in the grpc -tls cert of peers is usually include "." , for example, "peer1.org@example.com"
but, the service name do not include "." ,so that the tls verify bad
luckydogchina (Wed, 19 Sep 2018 10:19:58 GMT):
I find that host names in the grpc -tls cert of peers is usually include "." , for example, "peer1.org@example.com"
but, the service name do not include "." in k8s ,so that the tls hostname verify bad
luckydogchina (Wed, 19 Sep 2018 10:19:58 GMT):
I find that host names in the grpc -tls cert of peers is usually include "." , for example, "peer1.org@example.com"
but, the service name can not include "." in k8s ,so that the tls hostname verify bad, when some pods connecting to "peer1.org@example.com"
luckydogchina (Wed, 19 Sep 2018 10:19:58 GMT):
I find that host names in the grpc -tls cert of peers is usually include "." , for example, "peer1.org@example.com"
but, the service name can not include "." in k8s ,so that the tls hostname verify bad, when some pods connecting to "peer1.org@example.com"
how to resolve it?
luckydogchina (Wed, 19 Sep 2018 10:19:58 GMT):
I find that host names in the grpc -tls cert of peers usually include "." , for example, "peer1.org@example.com"
but, the service name can not include "." in k8s ,so that the tls hostname verify bad, when some pods connecting to "peer1.org@example.com"
how to resolve it?
luckydogchina (Wed, 19 Sep 2018 10:19:58 GMT):
I find that host names in the grpc -tls cert of peers usually include "." , for example, "peer1.org@example.com"
but, the service name can not include "." in k8s ,so that the tls hostname verified failure, when some pods connecting to "peer1.org@example.com"
how to resolve it?
julian (Fri, 21 Sep 2018 20:43:57 GMT):
Hello. We have been using Fabric on Docker swarm, but I now have some time to look at K8 as an option for us. I have played around with it via Minikube, but have hit an issue with pods not being able to hit their own service. e.g. a peer talking to itself via it's service. It appears to be a hairpin related issue on the virtual device, but moving forward, we don't intend to use Minikube for production, so it's probably not worth spending time to find a fix, unless someone has one to hand ;-) There seem to be lots of options for installing a cluster. We are looking primarily to use it on AWS, and maybe Azure. Can anyone recommend an approach to K8 with Fabric in mind?
Wizzy123 (Sun, 23 Sep 2018 13:45:35 GMT):
Has joined the channel.
Wizzy123 (Sun, 23 Sep 2018 13:45:40 GMT):
I have a doubt, as I am working on Hyperledger fabric
So
How to deploy chaincode in multiple organisation environment,
where all organisations in the environment has a peer node each,
so that any transaction that happens on single organization peer is reflected on the peers of other organizations
i.e how to keep the peers in each organization in sync ?
Or how to setup Hyperledger fabric in multiple organization environment ?
PrakharShukla (Mon, 24 Sep 2018 19:18:05 GMT):
Has joined the channel.
kjroger94 (Tue, 25 Sep 2018 02:42:31 GMT):
unable to start kafka, getting this error when doing it on k8s
kjroger94 (Tue, 25 Sep 2018 02:42:32 GMT):
`try -> DEBU 13c2 [channel: mychannel] Need to retry because process failed = kafka server: In the middle of a leadership election, there is currently no lead
er for this partition and hence it is unavailable for writes.
2018-09-25 02:22:19.707 UTC [orderer/consensus/kafka] startThread -> CRIT 13c3 [channel: mychannel] Cannot post CONNECT message = kafka server: In the middle of a leadership election, there is currently no leade
r for this partition and hence it is unavailable for writes.
panic: [channel: mychannel] Cannot post CONNECT message = kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes.`
rezmuh (Tue, 25 Sep 2018 09:38:30 GMT):
Has joined the channel.
usmanbinyahya (Tue, 25 Sep 2018 09:50:02 GMT):
Has joined the channel.
alexvicegrab (Tue, 25 Sep 2018 14:14:00 GMT):
Hello, we are doing a webinar with the LI on deploying with Helm Charts:
https://medium.com/aid-tech/webinar-a-hitchhikers-guide-to-deploying-hyperledger-fabric-on-kubernetes-1bcecc2a4ade
In case anyone from the is interested in attending
aatkddny (Tue, 25 Sep 2018 14:33:22 GMT):
Anyone use dind? We just hit a problem with disk pressure despite the /var/lib/docker omg that box being gigantic.
It looks like it's writing to the root file system - is that expected behavior or are we missing a parameter somewhere?
aatkddny (Tue, 25 Sep 2018 14:33:22 GMT):
Anyone use dind? We just hit a problem with disk pressure despite the /var/lib/docker omg that box being gigantic.
It looks like it's writing to the root file system - is that expected behavior or are we missing a parameter somewhere?
EDIT: NM it looks like it is. Something to be aware of if your SAs set your root filesystem size to 1Gb with a "but nobody every uses that" comment.
aatkddny (Tue, 25 Sep 2018 14:33:22 GMT):
Anyone use dind? We just hit a problem with disk pressure despite the /var/lib/docker omg that box being gigantic.
It looks like it's writing to the root file system - is that expected behavior or are we missing a parameter somewhere?
EDIT: NM it looks like it is. Something to be aware of if your SAs set your root filesystem size to 1Gb with a "but nobody every uses that" comment. No wonder it worked on a local machine...
ascatox (Wed, 26 Sep 2018 07:40:43 GMT):
Hi All! Where can i found a good guide to use Kubernetes to install Fabric on my servers?Thanks in advance!
ColeBoudreau (Wed, 26 Sep 2018 13:01:30 GMT):
@PascalVerlinden There's a seminar today actually! The links just above, but here it is again: https://medium.com/aid-tech/webinar-a-hitchhikers-guide-to-deploying-hyperledger-fabric-on-kubernetes-1bcecc2a4ade
ColeBoudreau (Wed, 26 Sep 2018 13:01:30 GMT):
@ascatox There's a seminar today actually! The links just above, but here it is again: https://medium.com/aid-tech/webinar-a-hitchhikers-guide-to-deploying-hyperledger-fabric-on-kubernetes-1bcecc2a4ade
PascalVerlinden (Wed, 26 Sep 2018 13:01:30 GMT):
Has joined the channel.
EvansChang (Thu, 27 Sep 2018 02:37:00 GMT):
Has joined the channel.
ascatox (Thu, 27 Sep 2018 14:33:01 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=dMcg7tgNEN8tvdgti) @ColeBoudreau Thank you very much I'll try to attend this.
easeev (Fri, 28 Sep 2018 12:07:46 GMT):
Has joined the channel.
hayorov (Fri, 28 Sep 2018 12:38:48 GMT):
Has joined the channel.
zacpl (Fri, 28 Sep 2018 18:37:49 GMT):
Has joined the channel.
kjroger94 (Sun, 30 Sep 2018 03:54:53 GMT):
`Error: error getting endorser client for channel: endorser client failed to connect to liqvis-service-peer2:30210: failed to create new connection: context deadline exceeded` i am getting this error while trying to run commands. if i even put the IP of the service, it does not connect. But if i give that of the pods in /etc/hosts, it connects. Peer1 is able to communicate with Peer1 via the service and I am able to curl the service as well but from inside the pod, the service connected to it is unreachable.
hayorov (Sun, 30 Sep 2018 10:53:26 GMT):
Guys, I'm trying to run fabric network on kubernetes and looks like have an issue with any chaincode execution ... for example - trying to `peer channel getinfo -c ANY` and got `2018-09-30 10:50:43.335 UTC [endorser] ProcessProposal -> ERRO 1d7 [][a28ec38c] simulateProposal() resulted in chaincode name:"qscc" response status 500 for txid: a28ec38c7d9cfa6e63940173b454a9c93dafb4bf588f8ad1ccd11308977ff722` how can i check that docker VM with system chaincode works? any HOWTO? tricks about mounted docker socket? I'm using GKE cluster
hayorov (Sun, 30 Sep 2018 10:55:34 GMT):
in startup logs of peer i see `2018-09-30 10:42:13.846 UTC [qscc] Init -> INFO 0ff Init QSCC
2018-09-30 10:42:13.846 UTC [shim] func1 -> DEBU 100 [3deec813] Init get response status: 200
2018-09-30 10:42:13.846 UTC [shim] func1 -> DEBU 101 [3deec813] Init succeeded. Sending COMPLETED
2018-09-30 10:42:13.846 UTC [shim] triggerNextState -> DEBU 102 [3deec813] send state message COMPLETED
2018-09-30 10:42:13.846 UTC [chaincode] handleMessage -> DEBU 103 [3deec813] Fabric side handling ChaincodeMessage of type: COMPLETED in state ready
2018-09-30 10:42:13.846 UTC [chaincode] Notify -> DEBU 104 [3deec813] notifying Txid:3deec813-5a92-47ff-b066-785270d35b37, channelID:
2018-09-30 10:42:13.846 UTC [chaincode] Execute -> DEBU 105 Exit
2018-09-30 10:42:13.846 UTC [sccapi] deploySysCC -> INFO 106 system chaincode qscc/(github.com/hyperledger/fabric/core/scc/qscc) deployed
2018-09-30 10:42:13.846 UTC [nodeCmd] serve -> INFO 107 Deployed system chaincodes
2018-09-30 10:42:13.847 UTC [discovery/lifecycle] InstalledCCs -> DEBU 108 Returning []
2018-09-30 10:42:13.847 UTC [discovery] NewService -> INFO 109 Created with config TLS: false, authCacheMaxSize: 1000, authCachePurgeRatio: 0.750000`
hayorov (Sun, 30 Sep 2018 10:55:34 GMT):
in startup logs of peer i see ```2018-09-30 10:42:13.846 UTC [qscc] Init -> INFO 0ff Init QSCC
2018-09-30 10:42:13.846 UTC [shim] func1 -> DEBU 100 [3deec813] Init get response status: 200
2018-09-30 10:42:13.846 UTC [shim] func1 -> DEBU 101 [3deec813] Init succeeded. Sending COMPLETED
2018-09-30 10:42:13.846 UTC [shim] triggerNextState -> DEBU 102 [3deec813] send state message COMPLETED
2018-09-30 10:42:13.846 UTC [chaincode] handleMessage -> DEBU 103 [3deec813] Fabric side handling ChaincodeMessage of type: COMPLETED in state ready
2018-09-30 10:42:13.846 UTC [chaincode] Notify -> DEBU 104 [3deec813] notifying Txid:3deec813-5a92-47ff-b066-785270d35b37, channelID:
2018-09-30 10:42:13.846 UTC [chaincode] Execute -> DEBU 105 Exit
2018-09-30 10:42:13.846 UTC [sccapi] deploySysCC -> INFO 106 system chaincode qscc/(github.com/hyperledger/fabric/core/scc/qscc) deployed
2018-09-30 10:42:13.846 UTC [nodeCmd] serve -> INFO 107 Deployed system chaincodes
2018-09-30 10:42:13.847 UTC [discovery/lifecycle] InstalledCCs -> DEBU 108 Returning []
2018-09-30 10:42:13.847 UTC [discovery] NewService -> INFO 109 Created with config TLS: false, authCacheMaxSize: 1000, authCachePurgeRatio: 0.750000```
archit90 (Sun, 30 Sep 2018 11:44:32 GMT):
Has joined the channel.
bairathirahul (Sun, 30 Sep 2018 17:51:40 GMT):
Has joined the channel.
Dpkkmr (Mon, 01 Oct 2018 04:39:29 GMT):
Has joined the channel.
julian (Mon, 01 Oct 2018 16:43:09 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=SXHs7qcFQaGwNzeXW) @kjroger94 I had similar issue when using Minikube. I found a container within a pod was not able to connect to it's own service. This was evident when trying to join a channel. I got around this by setting the following prior to running peer commands: CORE_PEER_ADDRESS=0.0.0.0:7051
Dima (Tue, 02 Oct 2018 07:46:22 GMT):
Has joined the channel.
Bartb0 (Tue, 02 Oct 2018 10:52:47 GMT):
Has joined the channel.
MaddaliPadmaja (Wed, 03 Oct 2018 07:14:30 GMT):
Has joined the channel.
Dima (Thu, 04 Oct 2018 11:08:08 GMT):
Has left the channel.
ffiore81 (Thu, 04 Oct 2018 14:53:27 GMT):
Has joined the channel.
kago (Fri, 05 Oct 2018 21:02:18 GMT):
Has joined the channel.
qiangqinqq (Sat, 06 Oct 2018 07:35:30 GMT):
Has joined the channel.
OviiyaDominic (Tue, 09 Oct 2018 04:37:48 GMT):
Has joined the channel.
OviiyaDominic (Tue, 09 Oct 2018 04:39:07 GMT):
swarm
chandrakanthMamillapalli (Wed, 10 Oct 2018 00:09:42 GMT):
Has joined the channel.
alexvicegrab (Wed, 10 Oct 2018 13:56:05 GMT):
I’m currently working on adding the possibility of exposing the orderers to the internet via ingresses. I’ve gotten as far as setting up the ingresses (using the NGINX-ingress controller) to accept GRPC connections for the Orderer and Peer:
https://github.com/kubernetes/ingress-nginx/tree/master/docs/examples/grpc
Using internal addresses works fine and is all dandy where the whole network lives within a single Kubernetes cluster.
However, I’m seeing issues when trying to connect from the peer to the orderer:
If I include the full address in `configtx.yaml`, including `https://` procotol for the ingress, I see errors like this (actual address replaced with `example.com`):
```
2018-10-10 13:04:29.940 UTC [grpc] Printf -> DEBU 368 grpc: addrConn.createTransport failed to connect to {https://ord2.example.com:443 0
iramiller (Wed, 10 Oct 2018 16:55:45 GMT):
You can't specify the https scheme because the peer is coded to make a GRPC connection. More details starting at about line 160 of `github.com/hyperledger/fabric/core/comm/client.go`
iramiller (Wed, 10 Oct 2018 16:57:05 GMT):
You need to investigate kubernetes ingress for GRPC services... https://github.com/kubernetes/ingress-nginx/tree/master/docs/examples/grpc
alexvicegrab (Thu, 11 Oct 2018 09:12:32 GMT):
Hi @iramiller, thanks for the reply. That is exactly the example I used, the example for kubernetes ingress in `ingress-nginx`, and this works fine for their example.
My current hypothesis for the issue in the second case is that the fact that we have SSL/TLS throws the Peer client off, which does not expect to be connect to a protected service. I guess I need to start by getting mutual TLS working internally in the cluster and then try my luck with K8S Ingresses again.
aatkddny (Thu, 11 Oct 2018 13:05:42 GMT):
If you get mutual TLS working I'd be interested in hearing how you did it. I sorta gave up once I read about K8S requiring a reverse proxy in every node. Fortunately we have our own data center, but if we ever cloud hosted it I'd have to go back and look and I'm not looking forward to it.
alexvicegrab (Thu, 11 Oct 2018 13:53:20 GMT):
Hi @aatkddny. Yes, the documentation is rather sparse on many of these questions, unfortunately. If I succeed, I will try to ping back here.
waxer (Fri, 12 Oct 2018 12:31:36 GMT):
Has joined the channel.
saadinator (Sat, 13 Oct 2018 08:06:16 GMT):
Has joined the channel.
hotbydefault (Sat, 13 Oct 2018 11:25:44 GMT):
Has joined the channel.
SumanPapanaboina (Sun, 14 Oct 2018 15:36:39 GMT):
Has joined the channel.
mastersingh24 (Mon, 15 Oct 2018 11:03:54 GMT):
@alexvicegrab - the issue is likely the fact that the peer does not trust the issues of the TLS certificate being using by the orderer (technically I suppose being used by nginx). Did you include the root / intermediate CA which issued the TLS certificate in your orderer org's MSP in the the channel definition?
mastersingh24 (Mon, 15 Oct 2018 11:03:54 GMT):
@alexvicegrab - the issue is likely the fact that the peer does not trust the issuer of the TLS certificate being using by the orderer (technically I suppose being used by nginx). Did you include the root / intermediate CA which issued the TLS certificate in your orderer org's MSP in the the channel definition?
mastersingh24 (Mon, 15 Oct 2018 11:03:54 GMT):
@alexvicegrab - the issue is likely the fact that the peer does not trust the issuer of the TLS certificate being using by the orderer (technically I suppose being used by nginx). Did you include the root / intermediate CA which issued the TLS certificate in your orderer org's MSP in the the channel definition? The other possibility is that the TLS certificate does not have any SANS matching whichever hostname you are adding in your configtx.yaml
mastersingh24 (Mon, 15 Oct 2018 11:04:44 GMT):
I assume that you did not actually enable TLS on the actually orderer node(s)? Just at the ingess / nginx layer?
mastersingh24 (Mon, 15 Oct 2018 11:05:49 GMT):
How did you actually create the crypto material?
alexvicegrab (Mon, 15 Oct 2018 11:08:00 GMT):
Hi @mastersingh24, you are correct, I did not enable TLS on the actual orderer at that point (looking into getting this working now on my Helm charts). The NGINX TLS certificates to access all the ingresses were generated for each orderer and peer with Let's Encrypt, by using a Helm Chart (K8S application package) called `cert-manager`. I did not (nor could) include these TLS certificates as they are generated after launching the orderers (which require a genesis block containing these).
mastersingh24 (Mon, 15 Oct 2018 11:08:42 GMT):
OK
mastersingh24 (Mon, 15 Oct 2018 11:08:57 GMT):
Are using cryptogen at all?
alexvicegrab (Mon, 15 Oct 2018 11:09:00 GMT):
No
alexvicegrab (Mon, 15 Oct 2018 11:09:07 GMT):
Only the Fabric CA
mastersingh24 (Mon, 15 Oct 2018 11:09:22 GMT):
How did you create the MSP info for your orgs?
alexvicegrab (Mon, 15 Oct 2018 11:09:27 GMT):
I use that to create all the crypto material
alexvicegrab (Mon, 15 Oct 2018 11:09:27 GMT):
I use the Fabric CA that to create all the crypto material.
mastersingh24 (Mon, 15 Oct 2018 11:10:02 GMT):
ok
alexvicegrab (Mon, 15 Oct 2018 11:10:28 GMT):
Except of course the TLS material of the NGINX ingress, which uses Let's Encrypt as the CA
mastersingh24 (Mon, 15 Oct 2018 11:13:07 GMT):
So what you will need to do is to add the root and intermediate CAs for Let's Encrypt into the `tlscacerts` and/or `tlsintermediatecerts` directory for your orderer organization prior to running `configtxgen`. If you also plan to expose the peers as well, you should do the same for your peer orgs. And of course make sure that the hostname(s) you use for the orderer endpoint(s) in `configtx.yaml` match the external hostname for your ingress
alexvicegrab (Mon, 15 Oct 2018 11:14:03 GMT):
Cool, thanks @mastersingh24, I will attempt to do this and get back to you
mastersingh24 (Mon, 15 Oct 2018 11:14:17 GMT):
sounds good
mastersingh24 (Mon, 15 Oct 2018 11:14:42 GMT):
sorry for the delayed response ... I don't often check this particular channel :(
mastersingh24 (Mon, 15 Oct 2018 11:14:58 GMT):
(too many channels for my pour soul)
mastersingh24 (Mon, 15 Oct 2018 11:14:58 GMT):
(too many channels for my poor soul)
MohitJuneja (Tue, 16 Oct 2018 03:25:11 GMT):
Has joined the channel.
CarlosRL (Wed, 17 Oct 2018 17:44:31 GMT):
Has joined the channel.
CarlosRL (Wed, 17 Oct 2018 17:58:05 GMT):
Hi, I am deploying a hyperledger network inside k8s
CarlosRL (Wed, 17 Oct 2018 17:58:33 GMT):
I have a question, once a hyperledger fabric network is deployed with kafka as consensus algorithm inside a kubernetes cluster. Such network is configured with three kafka brokers within the configtx.yml. Is it possible add new kafka brokers to the deployed network?
mastersingh24 (Thu, 18 Oct 2018 10:22:46 GMT):
@CarlosRL - Yes ... just follow the Kakfa docs on how to add brokers to a cluster. You don't actually have to update the channel config as the brokers listed there are simply used for bootstrapping the embedded Kafka client.
CarlosRL (Thu, 18 Oct 2018 14:49:24 GMT):
@mastersingh24 thanks :thumbsup:
tchataigner (Thu, 18 Oct 2018 15:32:19 GMT):
Has joined the channel.
minollo (Thu, 18 Oct 2018 16:48:55 GMT):
Has joined the channel.
laurasp (Thu, 18 Oct 2018 17:05:19 GMT):
Has joined the channel.
plato (Fri, 19 Oct 2018 20:23:27 GMT):
Has joined the channel.
alexvicegrab (Fri, 19 Oct 2018 23:22:18 GMT):
@mastersingh24. I'd like to ask your advice again.
I've tried to work through the issue by trying to get the orderers to use TLS. For this I stopped using the ingresses, and simply enabled TLS on the orderers. As you suggested, I added the `tlscacerts` and `tlsintermediatecerts` to the Peer and Orderer admin MSP directories prior to running configtxgen.
I managed to create and join the channel on the peers, by adding the relevant options: `--tls` `--ordererTLSHostnameOverride` and `--cafile`. However, despite successfully joining the channel, the peer has similar trouble connecting to the orderer.
On the orderer side, I see this:
```
2018-10-19 23:14:40.513 UTC [grpc] Printf -> DEBU 60c grpc: Server.Serve failed to complete security handshake from "10.244.0.218:45782": tls: first record does not look like a TLS handshake
2018-10-19 23:14:41.313 UTC [grpc] Printf -> DEBU 60d grpc: Server.Serve failed to complete security handshake from "10.244.2.59:49300": tls: first record does not look like a TLS handshake
```
And on the peer side I see this:
```
2018-10-19 23:14:40.990 UTC [grpc] Printf -> DEBU ccd pickfirstBalancer: HandleSubConnStateChange: 0xc422f7e1d0, CONNECTING
2018-10-19 23:14:40.995 UTC [grpc] Printf -> DEBU cce grpc: addrConn.createTransport failed to connect to {ord2-hlf-ord.blockchain.svc.cluster.local:7050 0
alexvicegrab (Fri, 19 Oct 2018 23:36:27 GMT):
P.S. I've tried both with and without `CORE_PEER_TLS_CLIENTAUTHREQUIRED` option set to `true` and `false` in the peer.
uherr89 (Mon, 22 Oct 2018 13:30:32 GMT):
Has joined the channel.
MohammadObaid (Tue, 23 Oct 2018 08:29:31 GMT):
Does anywhere here deploy fabric using only swarm network manually ?
mhs22 (Tue, 23 Oct 2018 09:36:14 GMT):
Has joined the channel.
feitnomore (Tue, 23 Oct 2018 17:14:06 GMT):
Has joined the channel.
feitnomore (Tue, 23 Oct 2018 17:21:04 GMT):
http://www.feitoza.com.br/blockchain-kube.txt
feitnomore (Tue, 23 Oct 2018 17:21:20 GMT):
this is the log... its all on one file, but I've separated the invoke, the orderer, and the peers
mastersingh24 (Tue, 23 Oct 2018 18:41:06 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=33gvhmQb5b5szago7) @alexvicegrab Do you have `CORE_PEER_TLS_ENABLED=true`?
alexvicegrab (Tue, 23 Oct 2018 21:17:44 GMT):
@mastersingh24, yes, both `CORE_PEER_TLS_ENABLED=true` and `ORE_PEER_TLS_CLIENTAUTHREQUIRED` as either `true` or `false`
alexvicegrab (Tue, 23 Oct 2018 21:18:52 GMT):
For each of Peers and Orderers, I obtained their TLS certificates from intermediate Fabric CAs (one for all Orderers, another for all Peers), which share a Root CA.
mastersingh24 (Wed, 24 Oct 2018 15:12:56 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=pQmyBMWAi4G38yEk5) @alexvicegrab One thing I forgot to mention earlier was that you need to make sure that the TLS certificates have a CommonName or a SAN (Subject Alternative Name) matching the hostname you use to address them in the cluster. There is no hostname override available for peer to peer or peer to orderer communication
alexvicegrab (Wed, 24 Oct 2018 17:07:13 GMT):
Aha... OK, that is very useful to know, thank you @mastersingh24. I will have a go at checking the certificates and re-issuing them if necessary.
mastersingh24 (Wed, 24 Oct 2018 18:43:28 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=fYxreWKMNct7Cqx44) @alexvicegrab Apologies for not mentioning that the first time
zimabry (Thu, 25 Oct 2018 01:15:10 GMT):
Has joined the channel.
cagdast (Thu, 25 Oct 2018 07:43:51 GMT):
Has joined the channel.
ArpitKhurana1 (Sat, 27 Oct 2018 07:04:07 GMT):
Has joined the channel.
ArpitKhurana1 (Sat, 27 Oct 2018 07:09:05 GMT):
Has anyone been able to use cello with kubernetes ( on gcp or azure)? I am having difficulty in adding kubernetes as host. Although I deployed fabric on kubernetes with my own configuration, but its really messy to maintain that
yousaf (Sun, 28 Oct 2018 19:02:48 GMT):
Has joined the channel.
yousaf (Sun, 28 Oct 2018 19:03:59 GMT):
Hi everyone. Which is the best approach to deploy fabric network on multiple hosts? Kubernetes or Docker Swarm?
MohammadObaid (Mon, 29 Oct 2018 09:56:55 GMT):
@mastersingh24 I have setup swarm network on 4 machines for fabric and successfully created channels and all other peers able to join channels. When I am trying to instantiate chaincode , it doesnt spinup ccenv container thus resulting in timeout failure . ccenv image present in a system but not spinning up any container . What could be possible reasons for that ? I ain't getting any errors in peer logs . I increased timeout to much larger value but nothing works so far.
mastersingh24 (Mon, 29 Oct 2018 11:15:34 GMT):
@MohammadObaid - is it launching ccenv but then not able to either build or launch the actual chaincode container?
MohammadObaid (Mon, 29 Oct 2018 11:43:47 GMT):
No it is not launching ccenv . I aint getting any ccenv stopped container . cli just wait and then throw timeout error if it hits timeout value
MohammadObaid (Mon, 29 Oct 2018 11:43:47 GMT):
No it is not launching ccenv . I aint getting any ccenv dev image like (dev-peerfirst-mycc-v0) neither any stopped container . cli just wait and then throw timeout error if it hits timeout value . Should'nt I get any errors on peer logs except timeout error ?
mastersingh24 (Mon, 29 Oct 2018 12:22:00 GMT):
what's the CLI error?
MohammadObaid (Mon, 29 Oct 2018 16:12:38 GMT):
@mastersingh24 This is the cli logs when I set timeout value to 3 minutes `- CORE_CHAINCODE_STARTUPTIMEOUT=180s`
MohammadObaid (Mon, 29 Oct 2018 16:13:08 GMT):
chaincodecli.png
MohammadObaid (Mon, 29 Oct 2018 16:13:52 GMT):
These are the peer logs
MohammadObaid (Mon, 29 Oct 2018 16:14:24 GMT):
chaincodepeerlog1.png
MohammadObaid (Mon, 29 Oct 2018 16:15:37 GMT):
chaincodepeerlog2.png
waxer (Tue, 30 Oct 2018 20:22:57 GMT):
@MohammadObaid check that the docker network name is configured propperly.
MohammadObaid (Wed, 31 Oct 2018 05:23:56 GMT):
@waxer where in peer container files or in orderer container.yaml files ? All my services are running on same network .
MohammadObaid (Wed, 31 Oct 2018 05:24:29 GMT):
Do I manually need to run ccenv container and link it with peer ?
MohammadObaid (Wed, 31 Oct 2018 11:46:45 GMT):
Hey @mastersingh24 any feedback from your side ? I am still stuck in same issue :(
waxer (Wed, 31 Oct 2018 11:51:41 GMT):
@MohammadObaid , I mean the CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE env variable of the peer. Is correctly set to the docker-compose network name of your container?
mastersingh24 (Wed, 31 Oct 2018 16:30:55 GMT):
You definitely need to set the ` CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE` to the name of the Docker overlay network being used by the Swarm cluster. You also need to make sure you mark that network as "attachable" else only containers started via Swarm will be able to connect to the network
MohammadObaid (Thu, 01 Nov 2018 11:04:34 GMT):
Hey @waxer @mastersingh24 thanks a lot for helping me out . Those two things were the missing part in my network . After making network attachable and including that environmental variable , I am able to invoke chaincode :)
m.hago (Thu, 01 Nov 2018 13:13:35 GMT):
Has joined the channel.
MohammedR (Thu, 01 Nov 2018 15:08:11 GMT):
Has joined the channel.
ArpitKhurana1 (Fri, 02 Nov 2018 04:00:46 GMT):
Hello everyone, has anyone tried making a channel between multiple kubernetes clusters ( single org or multiorg)?
ArpitKhurana1 (Fri, 02 Nov 2018 04:00:46 GMT):
Hello everyone, has anyone tried making a channel between multiple kubernetes clusters ( single org or multiorg)? I mean , like orderer and some peers in one cluster, and some other peers in another cluster
ArpitKhurana1 (Fri, 02 Nov 2018 04:00:46 GMT):
Hello everyone, has anyone tried making a channel between multiple kubernetes clusters ( single org or multiorg)? I mean , like orderer and some peers in one cluster, and some other peers in another cluster
ArpitKhurana1 (Fri, 02 Nov 2018 04:00:46 GMT):
Hello everyone, has anyone tried making a channel between multiple kubernetes clusters ( single org or multiorg)? I mean , like orderer and some peers in one cluster, and some other peers in another cluster```
I had few confusions regarding this,
```
ArpitKhurana1 (Fri, 02 Nov 2018 04:00:46 GMT):
Hello everyone, has anyone tried making a channel between multiple kubernetes clusters ( single org or multiorg)? I mean , like orderer and some peers in one cluster, and some other peers in another cluster
I had few confusions regarding this, like which peers need to have public ip and how the gossip protocol will work in this case
ddhulla (Mon, 05 Nov 2018 10:06:24 GMT):
Has joined the channel.
ddhulla (Mon, 05 Nov 2018 10:26:56 GMT):
Panic: runtime error: invalid memory address or nil pointer dereference while setting up fabric env
ddhulla (Mon, 05 Nov 2018 10:26:56 GMT):
Panic: runtime error: invalid memory address or nil pointer dereference while setting up fabric env
ddhulla (Mon, 05 Nov 2018 10:27:33 GMT):
While setting up the Fabric Env. on Kubernetes.
Cryptogen Starts
total 20
drwxr-xr-x. 3 1001 99 4096 Nov 3 12:55 chaincode
-rw-r--r--. 1 1001 99 7827 Nov 3 12:45 configtx.yaml
-rw-r--r--. 1 1001 99 4433 Nov 3 12:44 crypto-config.yaml
org1.main.tcloud.kpn.org
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x6a2280]
goroutine 1 [running]:
github.com/hyperledger/fabric/common/tools/cryptogen/msp.GenerateVerifyingMSP(0xc4202640c0, 0x23, 0x0, 0x0, 0xc420026900, 0x6, 0x0)
/opt/gopath/src/github.com/hyperledger/fabric/common/tools/cryptogen/msp/generator.go:194 +0x1b0
main.generatePeerOrg(0xac36b6, 0xd, 0xc4200263a8, 0x4, 0x0, 0x0, 0x0, 0xac0947, 0x2, 0xc420026970, ...)
/opt/gopath/src/github.com/hyperledger/fabric/common/tools/cryptogen/main.go:529 +0x8ce
main.generate()
/opt/gopath/src/github.com/hyperledger/fabric/common/tools/cryptogen/main.go:387 +0x139
main.main()
/opt/gopath/src/github.com/hyperledger/fabric/common/tools/cryptogen/main.go:221 +0x256
I'm referring below link to set up: https://github.com/IBM/blockchain-network-on-kubernetes
ssaddem (Mon, 05 Nov 2018 13:48:31 GMT):
hello
I have a problem : i want to migrate my data from docker compose to kubernate but the orderer address is hard writed in the block :
OrdererAddresses6
orderer.example.com:7050/Channel/Orderer/Admins""
and kubernate doesn't accept points in the service name and port less than the range "30000" any one can help me please ?
aatkddny (Mon, 05 Nov 2018 15:18:28 GMT):
you could add a kubernetes service
```
apiVersion: v1
kind: Service
metadata:
name: orderer0
spec:
type: NodePort
ports:
- name: external-endpoint
protocol: TCP
nodePort: 30020
port: 7050
targetPort: 7050
selector:
app: orderer0-orderer```
ssaddem (Mon, 05 Nov 2018 17:35:33 GMT):
my problem is when the peer call orderer.example.com:7050 he can't found him
aatkddny (Mon, 05 Nov 2018 18:20:52 GMT):
what's the orderer called inside k8s? iirc the . isn't valid for a name.
ssaddem (Mon, 05 Nov 2018 22:35:57 GMT):
Clipboard - November 5, 2018 11:35 PM
ssaddem (Mon, 05 Nov 2018 22:39:29 GMT):
this picture is from block file he is used from docker compose but when i want move my data to a kubernates deplyment peer get the block file from orderer but he try to connect to orderer.example.com
in kubernate i have to force the cluster to listen below 30000 and in service he can't accept "." in name of service
ddhulla (Tue, 06 Nov 2018 09:42:09 GMT):
While instantiating chain code on Kubernetes using non-root user it is giving below error.
ddhulla (Tue, 06 Nov 2018 09:42:11 GMT):
2018-11-06 09:21:59.564 UTC [chaincodeCmd] checkChaincodeCmdParams -> INFO 001 Using default escc
2018-11-06 09:21:59.564 UTC [chaincodeCmd] checkChaincodeCmdParams -> INFO 002 Using default vscc
Error: Error endorsing chaincode: rpc error: code = Unknown desc = error starting container: Post http://unix.sock/containers/create?name=nid1-org1peer1-cc-1.0: dial unix /host/var/run/docker.sock: connect: no such file or directory
Usage:
peer chaincode instantiate [flags]
Flags:
-C, --channelID string The channel on which this command should be executed
--collections-config string The file containing the configuration for the chaincode's collection
-c, --ctor string Constructor message for the chaincode in JSON format (default "{}")
-E, --escc string The name of the endorsement system chaincode to be used for this chaincode
-l, --lang string Language the chaincode is written in (default "golang")
-n, --name string Name of the chaincode
-P, --policy string The endorsement policy associated to this chaincode
-v, --version string Version of the chaincode specified in install/instantiate/upgrade commands
-V, --vscc string The name of the verification system chaincode to be used for this chaincode
Global Flags:
--cafile string Path to file containing PEM-encoded trusted certificate(s) for the ordering endpoint
--certfile string Path to file containing PEM-encoded X509 public key to use for mutual TLS communication with the orderer endpoint
--clientauth Use mutual TLS when communicating with the orderer endpoint
--keyfile string Path to file containing PEM-encoded private key to use for mutual TLS communication with the orderer endpoint
--logging-level string Default logging level and overrides, see core.yaml for full syntax
-o, --orderer string Ordering service endpoint
--ordererTLSHostnameOverride string The hostname override to use when validating the TLS connection to the orderer.
--tls Use TLS when communicating with the orderer endpoint
--transient string Transient map of arguments in JSON encoding
ddhulla (Tue, 06 Nov 2018 09:42:33 GMT):
Can anyone please guide how do we resolve this error?
mastersingh24 (Tue, 06 Nov 2018 10:52:06 GMT):
@ddhulla - in order to mount the Docker socket (`/host/var/run/docker.sock` ) from the host / worker node, you either need to be root or you need to be in the docker group. Assuming you are passing in a uid and gid, you should make sure that the GID corresponds to the host / worker node group which owns the Docker socket (the group is usually name `docker`)
smpakes (Tue, 06 Nov 2018 19:53:40 GMT):
Has joined the channel.
luckydogchina (Wed, 07 Nov 2018 02:47:44 GMT):
Hi, guys:
I deploy the fabric with kafka consensus plugin, but the kafka brokers are always restarting.
luckydogchina (Wed, 07 Nov 2018 02:47:55 GMT):
the error logs is this:
luckydogchina (Wed, 07 Nov 2018 02:49:14 GMT):
[2018-11-07 02:46:48,684] INFO Opening socket connection to server 10.42.4.132/10.42.4.132:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-11-07 02:46:48,684] INFO Socket connection established to 10.42.4.132/10.42.4.132:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2018-11-07 02:46:48,789] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2018-11-07 02:46:48,790] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server 'zookeeper0:2181,zookeeper1:2181,zookeeper2:2181' with timeout of 36000 ms
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1233)
at org.I0Itec.zkclient.ZkClient.
luckydogchina (Wed, 07 Nov 2018 02:49:14 GMT):
'[2018-11-07 02:46:48,684] INFO Opening socket connection to server 10.42.4.132/10.42.4.132:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-11-07 02:46:48,684] INFO Socket connection established to 10.42.4.132/10.42.4.132:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2018-11-07 02:46:48,789] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2018-11-07 02:46:48,790] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server 'zookeeper0:2181,zookeeper1:2181,zookeeper2:2181' with timeout of 36000 ms
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1233)
at org.I0Itec.zkclient.ZkClient.
luckydogchina (Wed, 07 Nov 2018 02:49:14 GMT):
>'[2018-11-07 02:46:48,684] INFO Opening socket connection to server 10.42.4.132/10.42.4.132:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
>[2018-11-07 02:46:48,684] INFO Socket connection established to 10.42.4.132/10.42.4.132:2181, initiating session (org.apache.zookeeper.ClientCnxn)
>[2018-11-07 02:46:48,789] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2018-11-07 02:46:48,790] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server 'zookeeper0:2181,zookeeper1:2181,zookeeper2:2181' with timeout of 36000 ms
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1233)
at org.I0Itec.zkclient.ZkClient.
luckydogchina (Wed, 07 Nov 2018 02:49:14 GMT):
>'[2018-11-07 02:46:48,684] INFO Opening socket connection to server 10.42.4.132/10.42.4.132:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
>[2018-11-07 02:46:48,684] INFO Socket connection established to 10.42.4.132/10.42.4.132:2181, initiating session (org.apache.zookeeper.ClientCnxn)
>[2018-11-07 02:46:48,789] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
>[2018-11-07 02:46:48,790] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server 'zookeeper0:2181,zookeeper1:2181,zookeeper2:2181' with timeout of 36000 ms
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1233)
at org.I0Itec.zkclient.ZkClient.
luckydogchina (Wed, 07 Nov 2018 02:49:56 GMT):
and the zookeeper logs :
luckydogchina (Wed, 07 Nov 2018 02:49:56 GMT):
# and the zookeeper logs
luckydogchina (Wed, 07 Nov 2018 02:51:22 GMT):
`2018-11-07 02:50:46,762 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x4e0 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,763 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,763 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,764 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:51,768 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 2 at election address zookeeper1/10.42.4.132:3888
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
at java.lang.Thread.run(Thread.java:748)
2018-11-07 02:50:51,768 [myid:1] - INFO [WorkerSender[myid=1]:QuorumPeer$QuorumServer@149] - Resolved hostname: zookeeper1 to address: zookeeper1/10.42.4.132
2018-11-07 02:50:51,768 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x4e0 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:51,769 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)`
luckydogchina (Wed, 07 Nov 2018 02:51:22 GMT):
'2018-11-07 02:50:46,762 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x4e0 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,763 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,763 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,764 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:51,768 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 2 at election address zookeeper1/10.42.4.132:3888
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
at java.lang.Thread.run(Thread.java:748)
2018-11-07 02:50:51,768 [myid:1] - INFO [WorkerSender[myid=1]:QuorumPeer$QuorumServer@149] - Resolved hostname: zookeeper1 to address: zookeeper1/10.42.4.132
2018-11-07 02:50:51,768 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x4e0 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:51,769 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)'
luckydogchina (Wed, 07 Nov 2018 02:51:22 GMT):
'2018-11-07 02:50:46,762 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x4e0 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,763 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,763 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,764 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:51,768 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 2 at election address zookeeper1/10.42.4.132:3888
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
at java.lang.Thread.run(Thread.java:748)
2018-11-07 02:50:51,768 [myid:1] - INFO [WorkerSender[myid=1]:QuorumPeer$QuorumServer@149] - Resolved hostname: zookeeper1 to address: zookeeper1/10.42.4.132
2018-11-07 02:50:51,768 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x4e0 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:51,769 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)'
'''
luckydogchina (Wed, 07 Nov 2018 02:51:22 GMT):
'2018-11-07 02:50:46,762 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x4e0 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,763 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,763 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,764 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:51,768 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 2 at election address zookeeper1/10.42.4.132:3888
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
at java.lang.Thread.run(Thread.java:748)
2018-11-07 02:50:51,768 [myid:1] - INFO [WorkerSender[myid=1]:QuorumPeer$QuorumServer@149] - Resolved hostname: zookeeper1 to address: zookeeper1/10.42.4.132
2018-11-07 02:50:51,768 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x4e0 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:51,769 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)'
'''
luckydogchina (Wed, 07 Nov 2018 02:51:22 GMT):
'''2018-11-07 02:50:46,762 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x4e0 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,763 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,763 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,764 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:51,768 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 2 at election address zookeeper1/10.42.4.132:3888
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
at java.lang.Thread.run(Thread.java:748)
2018-11-07 02:50:51,768 [myid:1] - INFO [WorkerSender[myid=1]:QuorumPeer$QuorumServer@149] - Resolved hostname: zookeeper1 to address: zookeeper1/10.42.4.132
2018-11-07 02:50:51,768 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x4e0 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:51,769 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)'
'''
luckydogchina (Wed, 07 Nov 2018 02:51:22 GMT):
2018-11-07 02:50:46,762 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x4e0 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,763 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,763 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,764 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:51,768 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 2 at election address zookeeper1/10.42.4.132:3888
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
at java.lang.Thread.run(Thread.java:748)
2018-11-07 02:50:51,768 [myid:1] - INFO [WorkerSender[myid=1]:QuorumPeer$QuorumServer@149] - Resolved hostname: zookeeper1 to address: zookeeper1/10.42.4.132
2018-11-07 02:50:51,768 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x4e0 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:51,769 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)'
'''
luckydogchina (Wed, 07 Nov 2018 02:51:22 GMT):
>>2018-11-07 02:50:46,762 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x4e0 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,763 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,763 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,764 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:51,768 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 2 at election address zookeeper1/10.42.4.132:3888
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
at java.lang.Thread.run(Thread.java:748)
2018-11-07 02:50:51,768 [myid:1] - INFO [WorkerSender[myid=1]:QuorumPeer$QuorumServer@149] - Resolved hostname: zookeeper1 to address: zookeeper1/10.42.4.132
2018-11-07 02:50:51,768 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x4e0 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:51,769 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)'
'''
luckydogchina (Wed, 07 Nov 2018 02:51:22 GMT):
>2018-11-07 02:50:46,762 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x4e0 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
>2018-11-07 02:50:46,763 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
>2018-11-07 02:50:46,763 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2018-11-07 02:50:46,764 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
>2018-11-07 02:50:51,768 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 2 at election address zookeeper1/10.42.4.132:3888
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
at java.lang.Thread.run(Thread.java:748)
2018-11-07 02:50:51,768 [myid:1] - INFO [WorkerSender[myid=1]:QuorumPeer$QuorumServer@149] - Resolved hostname: zookeeper1 to address: zookeeper1/10.42.4.132
>2018-11-07 02:50:51,768 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x4e0 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
>2018-11-07 02:50:51,769 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x4df (n.round), LEADING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)'
'''
luckydogchina (Wed, 07 Nov 2018 02:59:21 GMT):
I set 6000 to *ZOO_TICKET_TIME*
luckydogchina (Wed, 07 Nov 2018 02:59:21 GMT):
I set 6000 to *ZOO_TICKET_TIME* , but not solve the error.
paul.sitoh (Wed, 07 Nov 2018 09:06:26 GMT):
Anyone know if there are any write-ups on how to deploy fabric to Kube? Something easy to read but sufficiently technical?
enriquebusti (Wed, 07 Nov 2018 12:01:27 GMT):
Has joined the channel.
matthewphamilton (Wed, 07 Nov 2018 15:22:17 GMT):
Has joined the channel.
yousaf (Thu, 08 Nov 2018 08:55:10 GMT):
Hi everyone. Is there any way that we could make a downloadable image for each peer that can be installed on each peer so they become part of the blockchain network and these peers can then communicate in a fabric network?
feitnomore (Fri, 09 Nov 2018 13:01:31 GMT):
Hello, I've created a Hyperledger Fabric Network over Kubernetes with 4 Orgs, 1 Peer each, + 1 orderer, and I'm trying to use a policy -P "AND('Org1.member','Org2.member','Org3.member','Org4.member')" but I'm not having success. People told me I would not be able to fire a transaction from command line for a policy like that because I need to collect endorsement from every organization, so I've tried with Node SDK, but it is still not working. Any help would be much appreciated. I have all the config files and log files available if needed.
AlexanderZhovnuvaty (Mon, 12 Nov 2018 11:06:02 GMT):
Has left the channel.
NicolasHuray (Mon, 12 Nov 2018 22:00:15 GMT):
Has joined the channel.
Switch2Logic (Tue, 13 Nov 2018 14:15:25 GMT):
Has left the channel.
aatkddny (Tue, 13 Nov 2018 14:47:46 GMT):
Need a sanity check. I'm thinking about TLS - which was my first mistake.
To get into a peer (or orderer or whatever) I ~think~ I can use nginx as an ingress - setting an image as a pod in each of the deployments and handling the SSL ugliness there.
But will that mean I need to expose a nodeport for each deployment, or is there a more elegant way to do this?
aatkddny (Tue, 13 Nov 2018 14:47:46 GMT):
Need a sanity check. I'm thinking about TLS - which was my first mistake.
To get into a peer (or orderer or whatever) I *think* I can use nginx as an ingress - setting an image as a pod in each of the deployments and handling the SSL ugliness there.
But will that mean I need to expose a nodeport for each deployment, or is there a more elegant way to do this?
midoblgsm (Tue, 13 Nov 2018 17:24:19 GMT):
Has joined the channel.
midoblgsm (Tue, 13 Nov 2018 17:33:17 GMT):
hi feitnomore, I am new to hyperledger, I see that you were successful in deploying hyperledger network, could you please share the configtx.yaml and crypto-config.yaml files that you used?
midoblgsm (Tue, 13 Nov 2018 17:33:43 GMT):
I succeeded creating the network but I could not add a channel with hyperledger 1.3.0
midoblgsm (Tue, 13 Nov 2018 17:34:03 GMT):
adding the channel I am getting this error:
midoblgsm (Tue, 13 Nov 2018 17:34:08 GMT):
```Error: got unexpected status: BAD_REQUEST -- error authorizing update: error validating DeltaSet: policy for [Group] /Channel/Application not satisfied: Failed to reach implicit threshold of 1 sub-policies, required 1 remaining```
ArpitKhurana1 (Wed, 14 Nov 2018 17:27:45 GMT):
Hey guys, how Can we maintain the state of orderers and peers (so that if they crash, they can come back in same state). I know that couchdb data and ledger data(production folder) are to persisted. What else steps are required? Do i need to make the peer join channel again, do i need to install chaincode again?
kevinkbc (Wed, 14 Nov 2018 20:22:03 GMT):
Has joined the channel.
sh777 (Thu, 15 Nov 2018 02:51:07 GMT):
Has joined the channel.
h4995974 (Thu, 15 Nov 2018 22:04:04 GMT):
Has joined the channel.
h4995974 (Thu, 15 Nov 2018 22:04:05 GMT):
Hello everyone.
We're receiving the following error when trying to spin up peers inside a kubernetes pods. `2018-11-15 21:52:25.172 UTC [gossip/gossip] handleMessage -> WARN 01f Message GossipMessage: tag:EMPTY alive_msg:
ArpitKhurana1 (Fri, 16 Nov 2018 07:09:16 GMT):
@h4995974 I dont think this is an error. Its just a gossip message
yousaf (Sun, 18 Nov 2018 07:56:06 GMT):
Hi everyone. Is it possible to deploy hyperledger fabric network on multiple machines using minikube?
yousaf (Sun, 18 Nov 2018 07:56:06 GMT):
Hi everyone. Is it possible to deploy hyperledger fabric network on multiple machines using minikube? Does minikube allow to use only one node or we can use it somehow for multiple nodes?
ArchanGanguly (Sun, 18 Nov 2018 15:30:05 GMT):
Has joined the channel.
sudhir.kumawat (Tue, 20 Nov 2018 07:40:01 GMT):
Hi All, I am getting error Failed to receive commit notification from peer while try to start business network with fabric version 1.3
RobertDiebels (Tue, 20 Nov 2018 08:27:12 GMT):
Has joined the channel.
greivinlopez (Wed, 21 Nov 2018 20:54:56 GMT):
Has joined the channel.
ArpitKhurana1 (Thu, 22 Nov 2018 07:19:38 GMT):
@yousaf I don't believe you can do that using minikube, you have to make a full kubernetes cluster,minikube is supposed to be for single node
VadimInshakov (Fri, 23 Nov 2018 13:41:54 GMT):
Has joined the channel.
VadimInshakov (Fri, 23 Nov 2018 13:42:17 GMT):
**Create channel error in Kubernetes job**
Hello!
I'm trying to create fabric network in kubernetes cluster using this samples: https://github.com/IBM/blockchain-network-on-kubernetes
But when I run create_channel.yaml, I see this error in logs:
```
2018-11-23 10:20:52.633 UTC [common/tools/configtxgen] main -> INFO 001 Loading configuration
2018-11-23 10:20:52.646 UTC [common/tools/configtxgen] doOutputChannelCreateTx -> INFO 002 Generating new channel configtx
2018-11-23 10:20:52.646 UTC [common/tools/configtxgen] main -> CRIT 003 Error on outputChannelCreateTx: config update generation failure: cannot define a new channel with no Application section
```
But really I have Application section and configtx.yaml works fine without kubernetes. Very strange.
my configtx.yaml: https://gist.github.com/VadimInshakov/fafb9788e53c19caf73ddb0b0228d00a#file-23434553yer
VadimInshakov (Fri, 23 Nov 2018 14:10:24 GMT):
create_channel.yaml: https://gist.github.com/VadimInshakov/7b9f8e614d55cbc8ceb1756df1564b36#file-create_channel-yaml
yousaf (Sun, 25 Nov 2018 17:28:01 GMT):
@ArpitKhurana1 Thanks sir :)
jiribroulik (Mon, 26 Nov 2018 15:43:52 GMT):
Has joined the channel.
jiribroulik (Mon, 26 Nov 2018 15:44:54 GMT):
Hello guys, does anyone know why my when I enable mutual tls I am getting this error?
jiribroulik (Mon, 26 Nov 2018 15:45:00 GMT):
'''2018-11-26 15:42:29.812 UTC [grpc] HandleSubConnStateChange -> DEBU 16de pickfirstBalancer: HandleSubConnStateChange: 0xc42320d330, CONNECTING
2018-11-26 15:42:29.835 UTC [grpc] createTransport -> DEBU 16df grpc: addrConn.createTransport failed to connect to {orderer1.alice.example.com:7050 0
jiribroulik (Mon, 26 Nov 2018 15:45:23 GMT):
the above is on peer
jiribroulik (Mon, 26 Nov 2018 15:45:29 GMT):
this is on orderer: remote error: tls: bad certificate
jiribroulik (Mon, 26 Nov 2018 15:47:21 GMT):
and my env variables for the peer on k8s are the following
jiribroulik (Mon, 26 Nov 2018 15:47:33 GMT):
env:
- name: FABRIC_CA_CLIENT_HOME
value: /etc/hyperledger/fabric
- name: FABRIC_CA_CLIENT_TLS_CERTFILES
value: /etc/hyperledger/fabric/tls/ca.crt
- name: CORE_LOGGING_GRPC
value: debug
- name: CORE_LEDGER_STATE_STATEDATABASE
value: CouchDB
- name: CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS
value: localhost:5984
- name: CORE_VM_ENDPOINT
value: unix:///host/var/run/docker.sock
- name: CORE_LOGGING_LEVEL
value: debug
- name: CORE_PEER_ADDRESSAUTODETECT
value: "false"
- name: CORE_PEER_TLS_ENABLED
value: "true"
- name: CORE_PEER_GOSSIP_USELEADERELECTION
value: "true"
- name: CORE_PEER_GOSSIP_ORGLEADER
value: "false"
- name: CORE_PEER_PROFILE_ENABLED
value: "true"
- name: CORE_PEER_MSPCONFIGPATH
value: /etc/hyperledger/fabric/msp
- name: CORE_PEER_TLS_CERT_FILE
value: /etc/hyperledger/fabric/tls/cert.crt
- name: CORE_PEER_TLS_KEY_FILE
value: /etc/hyperledger/fabric/tls/cert.key
- name: CORE_PEER_TLS_ROOTCERT_FILE
value: /etc/hyperledger/fabric/tls/ca.crt
- name: CORE_PEER_TLS_CLIENTAUTHREQUIRED
value: "true"
- name: CORE_PEER_TLS_CLIENTROOTCAS_FILES
value: /etc/hyperledger/fabric/tls/ca.crt
- name: CORE_PEER_TLS_CLIENTCERT_FILE
value: /etc/hyperledger/fabric/tls/admin.crt
- name: CORE_PEER_TLS_CLIENTKEY_FILE
value: /etc/hyperledger/fabric/tls/admin.key
- name: ORG_ADMIN_CERT
value: /etc/hyperledger/fabric/msp/admincerts/cert.pem
- name: CORE_PEER_ID
value: peer0.alice.example.com
- name: CORE_PEER_TLS_SERVERHOSTOVERRIDE
value: peer0.alice.example.com
- name: PEER_NAME
value: peer0.alice.example.com
- name: PEER_HOST
value: peer0.alice.example.com
- name: CORE_PEER_ADDRESS
value: $(CORE_PEER_ID):7051
- name: CORE_PEER_ADDRESS
value: $(CORE_PEER_ID):7051
- name: PEER_HOME
value: /etc/hyperledger/fabric
- name: ORG_NAME
value: b3663dfb-b9b2-47c3-aaf7-f8cc7846bd12
- name: CORE_PEER_GOSSIP_EXTERNALENDPOINT
value: $(CORE_PEER_ID):7051
- name: CORE_PEER_CHAINCODELISTENADDRESS
value: 0.0.0.0:7052
- name: CORE_PEER_LOCALMSPID
value: b3663dfb-b9b2-47c3-aaf7-f8cc7846bd12
- name: GODEBUG
value: netdns=go
jiribroulik (Mon, 26 Nov 2018 15:49:54 GMT):
if I run the following from the commandline from the peer I fetch the channel config with no problem peer channel fetch config config_block.pb -o orderer1.alice.example.com:7050 -c samplechannel10cccytgwm2pp-ywnmytbiogqw --tls --clientauth --cafile /etc/hyperledger/fabric/tls/ca.crt --keyfile /etc/hyperledger/fabric/tls/cert.key --certfile /etc/hyperledger/fabric/tls/cert.crt
JoanArcas (Tue, 27 Nov 2018 11:08:36 GMT):
Has joined the channel.
joaquimpedrooliveira (Tue, 27 Nov 2018 17:11:45 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=q95WWEhtku6yqtvCK) @ArpitKhurana1 You have to persist the contents of the following directories:
- Orderer: `/var/hyperledger/production/orderer/`
- Peers: `:/var/hyperledger/production`
We mapped these as persistent volume claims in K*S
joaquimpedrooliveira (Tue, 27 Nov 2018 17:11:45 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=q95WWEhtku6yqtvCK) @ArpitKhurana1 You have to persist the contents of the following directories:
- Orderer: `/var/hyperledger/production/orderer/`
- Peers: `:/var/hyperledger/production`
We mapped these as persistent volume claims in K8S
vanitas92 (Tue, 27 Nov 2018 17:53:52 GMT):
Hello guys! I have a question regarding a couchdb instances needed for production scenario in K8S. We currently have 2 peer x organization but only one instance of couchdb acting as ledger x organization. Both peers connect to the same couchdb instance. Is this approach correct or each peer should have its own couchdb instance? Thank you guys!
dave.enyeart (Tue, 27 Nov 2018 22:01:45 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=KJMdh5QFPx6DiZQbg) @vanitas92 You must have a 1:1 relation between peer and couchdb
ArpitKhurana1 (Wed, 28 Nov 2018 05:03:12 GMT):
@joaquimpedrooliveira Thanks for the answer. Are you sure that is it, I saw a ppt where a number of steps were written, like fetch the certificates again from ca, join the channel ,install the chaincode etc
ArpitKhurana1 (Wed, 28 Nov 2018 05:04:07 GMT):
Personally i have tried your way earlier , it worked most of times but not everytime
yousaf (Wed, 28 Nov 2018 12:41:34 GMT):
Hi everyone. I have set CORE_MSP_CONFIG PATH to /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/example.com/users/Admin@example.com/msp ..............in my peer deployment file. But when the pod is created for this peer, it keeps on crashing and I have checked the logs of this pod and it says that: Cannot run peer because cannot init crypto, missing /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/example.com/users/Admin@example.com/msp folder
Can anyone tell me the solution of this?
feitnomore (Wed, 28 Nov 2018 21:01:22 GMT):
I've created a little guide on my adventures running Fabric on Kubernetes
feitnomore (Wed, 28 Nov 2018 21:01:26 GMT):
if anyone is interested:
feitnomore (Wed, 28 Nov 2018 21:01:27 GMT):
https://github.com/feitnomore/hyperledger-fabric-kubernetes
knagware9 (Thu, 29 Nov 2018 07:05:51 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=oJB4XHo6snTadnePL) @feitnomore Thanks ,,I want to try
haggis (Thu, 29 Nov 2018 08:08:10 GMT):
Has joined the channel.
TopJohn (Fri, 30 Nov 2018 06:00:07 GMT):
Has joined the channel.
TopJohn (Fri, 30 Nov 2018 06:02:50 GMT):
@feitnomore in fabric 1.3 you can use cli to invoke transaction use peeraddress to appoint the peers
labcoinpoc (Fri, 30 Nov 2018 08:22:42 GMT):
Has joined the channel.
feitnomore (Fri, 30 Nov 2018 12:52:38 GMT):
I was doing everything in 1.2
feitnomore (Fri, 30 Nov 2018 12:52:44 GMT):
decided to upgrade to 1.3 last week
feitnomore (Fri, 30 Nov 2018 12:52:51 GMT):
so , just adapted / tested some stuff
feitnomore (Fri, 30 Nov 2018 12:53:00 GMT):
had to fix some stuff on configtx.yaml and that was it
feitnomore (Fri, 30 Nov 2018 12:53:15 GMT):
working on having kafka + zookeeper inside kubernetes next
waxer (Sat, 01 Dec 2018 09:41:42 GMT):
@feitnomore , great doc.
The only thing that makes me wonder if it is a good idea to have 3 replicas of an Orderen behind a Service.
Those orderers may eventually by out of sync in the block height, and I'm not sure if the balancing between them of the service could lead to some unexpected situation to a leader peer. Since the peer has been notified about block number X, and then maybe latter if the service chooses another replica that could have the latest block be X-1.
waxer (Sat, 01 Dec 2018 09:47:29 GMT):
Im not aware if the implementation details, but i guess it shouldn't be an issue since the 'deliver' interface asks always for newer blocks from peer POV.
waxer (Sat, 01 Dec 2018 09:47:44 GMT):
Maybe someone could check this
CarlosRL (Sun, 02 Dec 2018 16:27:05 GMT):
Hi, I'm running hyperledger fabric on kubernetes for the past couple o months. Now, I getting this error on my peer
```
2018-12-02 03:22:10.803 UTC [blocksProvider] DeliverBlocks -> WARN 18c4de [mychannel] Got error &{SERVICE_UNAVAILABLE}
2018-12-02 03:22:11.942 UTC [gossip/election] waitForInterrupt -> DEBU 18c4df [126 56 70 115 1 124 77 156 35 129 83 228 71 163 254 184 157 127 158 164 164 149 190 209 50 21 248 106 145 255 191 121] : Exiting
```
on the orderer
```
2018-12-02 03:44:12.198 UTC [common/deliver] deliverBlocks -> WARN 1375 [channel: mychannel] Rejecting deliver request for 10.64.1.6:40506 because of consenter error
```
By reading the hyperledger FAQ's this is due an error on the kafka/zookeeper
kafka logs show the following error
```
[2018-12-02 03:53:08,050] ERROR [ReplicaFetcherThread-0-1], Current offset 0 for partition [testchainid,0] out of range; reset offset to 4010 (kafka.server.ReplicaFetcherThread)
```
So, I changed offset auto reset to smallest, on the zookeeper by setting an environment variable. I didn't have luck. Any suggestions, thanks in advance
yousaf (Sun, 02 Dec 2018 20:55:26 GMT):
Hi everyone. I have a query that If a pod containing peer fails in Kubernetes and as scalling feature of Kubernetes allows automatic healing capability. So if a pod crashes and recreated by Kubernetes then Is there going to occur any change in the certificates assigned to that peer OR Is there going to be any effect on the chaincode installed on that peer I mean is there any possibilty that our network crashes because of any of these cases? Actually I want to understand that how the Fabric network is going to behave in Kubernetes if some of it component's pod or container crashes?
waxer (Mon, 03 Dec 2018 02:04:46 GMT):
@yousaf, the most important thing is that when the pod is recreated the pod.can have all the necessary artifacts as it had when you ran it he first time. Dont worry about the ledger or world state because it will reprocess all the blocks again.
waxer (Mon, 03 Dec 2018 02:05:32 GMT):
As you say, consider as 'artifact' the chaincodes too...
waxer (Mon, 03 Dec 2018 02:06:59 GMT):
Im running Fabric without k8 and I take this same considerations. I want to be easy to blow up VM instances and start from 0 easily
Taffies (Mon, 03 Dec 2018 05:13:46 GMT):
Hello! Did anyone use Fabric CA instead of cryptogen in Kubernetes? :)
yousaf (Mon, 03 Dec 2018 10:21:58 GMT):
@waxer Got it sir. Thanks :)
ihormudryy (Mon, 03 Dec 2018 10:23:57 GMT):
Has joined the channel.
yousaf (Mon, 03 Dec 2018 10:44:15 GMT):
I have another query. If we have deployed fabric on Google Kubernetes Engine then how to use each VM instance for each connected host to the cluster, as a peer of fabric network? and how each of that peer being a physical host is going to perform queries against chaincode w.r.t to its own rights.? Can somebody explain it briefly?
julian (Mon, 03 Dec 2018 11:58:21 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=e47d817b-0796-4a4d-a223-20033d83b443) @waxer We currently use persistence to ensure a peer state is maintained. If you don't use persistence, how are you ensuing a peer joins the channels it was part of after a restart?
julian (Mon, 03 Dec 2018 11:58:21 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=e47d817b-0796-4a4d-a223-20033d83b443) @waxer We currently use persistence to ensure a peers state is maintained. If you don't use persistence, how are you ensuing a peer joins the channels it was part of after a restart?
waxer (Mon, 03 Dec 2018 16:11:38 GMT):
@julian , if you consider the peers genesis block channels it belongs too as artifacts, you can make a init script that makes the joins.
arjitkhullar (Wed, 05 Dec 2018 00:02:22 GMT):
Has joined the channel.
julian (Wed, 05 Dec 2018 19:19:03 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=1b1094d5-9878-4d74-833a-d8bd2bf4142e) @waxer :thumbsup:
aatkddny (Fri, 07 Dec 2018 01:27:41 GMT):
Is there a way to stop kubernetes evicting a pod when it decides it's low on disk? because when it does it to one of the kafka in my network all hell breaks loose. i'd like to not go through this a third time now.
aatkddny (Fri, 07 Dec 2018 01:27:41 GMT):
Is there a way to stop kubernetes evicting a pod when it decides it's low on disk? because when it does it to one of the kafkas in my network all hell breaks loose. i'd like to not go through this a third time now.
RobertDiebels (Mon, 10 Dec 2018 10:29:20 GMT):
Afaik you can avoid that only by ensuring there is enough disk space. Also, are you sure it's due to disk space? I've had instances where the node wasn't responding within an expected time-frame which prompted the eviction. Setting a resource-limit on your containers might help resolve that issue.
aatkddny (Mon, 10 Dec 2018 13:29:48 GMT):
one was showing low imagefs and another low disk before it evicted. of course the desktop may not have been reporting accurately, but...
louisliu2048 (Tue, 11 Dec 2018 03:30:38 GMT):
Has joined the channel.
sivak2018 (Tue, 11 Dec 2018 04:30:05 GMT):
Has joined the channel.
thipuvaasan (Tue, 11 Dec 2018 04:43:20 GMT):
Has joined the channel.
aatkddny (Thu, 13 Dec 2018 01:03:52 GMT):
Need to wave my ignorance about a little as I'm doing some work with configuration and want to only do it once.
I have a question about chaincode instantiation.
We've been doing it using docker in docker, so each pod has a peer, a couchdb and a dind container.
It seems to be working fine.
We went that way because frankly the only documentation I saw where people had it working said they used dind, and when I figured out the
``` - name: CORE_VM_ENDPOINT
value: tcp://localhost:2375
```
trick it was pretty straightforward.
So time marches on and it's a bit more mainstream deploying in k8s. people must be hitting the it doesn't fit in a docker instance and swarm is incomprehensible to mere mortals problem.
I'm now seeing helm charts popping up all over the place.
Not a single one uses dind, instead it's all /var/run/docker.sock.
Is the accepted standard now to not use dind, but to go back to using the host?
And here's where I have to start showing my ignorance - istr that to get it to work one has to modify the docker config scripts as described here `http://www.think-foundry.com/deploy-hyperledger-fabric-on-kubernetes-part-1/`.
So a little more poking about has our friends at ibm doing it the same way we are `https://github.com/IBM/blockchain-network-on-kubernetes/blob/master/configFiles/docker.yaml`
Which doesn't help.
Is there an accepted standard, or is it just use what works?
If the latter I'll stick with dind - it's not proving to be a pain point.
aatkddny (Thu, 13 Dec 2018 01:03:52 GMT):
Need to wave my ignorance about a little as I'm doing some work with my auto configuration generation code and want to only do it once.
I have a question about chaincode instantiation.
We've been doing it using docker in docker, so each pod has a peer, a couchdb and a dind container.
It seems to be working fine.
We went that way because frankly the only documentation I saw where people had it working said they used dind, and when I figured out the
``` - name: CORE_VM_ENDPOINT
value: tcp://localhost:2375
```
trick it was pretty straightforward.
So time marches on and it's a bit more mainstream deploying in k8s. people must be hitting the it doesn't fit in a docker instance and swarm is incomprehensible to mere mortals problem.
I'm now seeing helm charts popping up all over the place.
Not a single one uses dind, instead it's all /var/run/docker.sock.
Is the accepted standard now to not use dind, but to go back to using the host?
And here's where I have to start showing my ignorance - istr that to get it to work one has to modify the docker config scripts as described here `http://www.think-foundry.com/deploy-hyperledger-fabric-on-kubernetes-part-1/`.
So a little more poking about has our friends at ibm doing it the same way we are `https://github.com/IBM/blockchain-network-on-kubernetes/blob/master/configFiles/docker.yaml`
Which doesn't help.
Is there an accepted standard, or is it just use what works?
If the latter I'll stick with dind - it's not proving to be a pain point.
varubasi77 (Thu, 13 Dec 2018 14:46:06 GMT):
Has joined the channel.
alexvicegrab (Sat, 15 Dec 2018 11:14:55 GMT):
We have done a workshop at HGF on deploying Fabric on Kubernetes in dev and prod:
https://hgf18.sched.com/event/b76c86de07c3bcaa094a8b149470e0e7
alexvicegrab (Sat, 15 Dec 2018 11:15:50 GMT):
We have released Fabric/Composer helm charts on https://hub.kubeapps.com/charts?q=hyperledger and aim to release a public library that automates the process using python. If anyone is interested in contributing, you are very welcome.
alexvicegrab (Sat, 15 Dec 2018 11:16:24 GMT):
The workshop slides are public, and so is the repository: https://github.com/aidtechnology/hgf-k8s-workshop
nicolapaoli (Sun, 16 Dec 2018 13:45:40 GMT):
Hi all, anyone here is developing Helm Charts or other solutions to deploy Hyperledger Projects (not only Fabric) on Kubernetes?
aatkddny (Sun, 16 Dec 2018 16:24:57 GMT):
We did it from a java app. Or more accurately wrote something inside the app to run the crypto and then generate the yaml from a configuration file.
DJ_HC (Sun, 16 Dec 2018 18:50:48 GMT):
Has joined the channel.
dinoradulovic (Mon, 17 Dec 2018 02:50:50 GMT):
Has joined the channel.
aviralwal (Mon, 17 Dec 2018 11:49:13 GMT):
Has joined the channel.
RobertDiebels (Mon, 17 Dec 2018 22:21:58 GMT):
@aatkddny Here's a list of reasons not to use DIND https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/ . Currently the unix-socket approach is also a hack until Fabric provides a means to bring chaincodes under the management of K8s afaik. There are some JIRA issues that provide more in depth information on what needs to be done for this but I don't have them at the ready right now.
RobertDiebels (Mon, 17 Dec 2018 22:24:36 GMT):
I released a npm package that takes care of the whole shabang about half a year ago. It generates the crypto files, creates the necessary k8s objects from that and uses HLF's config files to create the network. Only down side is it only deploys to GCE.
RobertDiebels (Mon, 17 Dec 2018 22:25:05 GMT):
It's called kubechain if you're interested.
aatkddny (Tue, 18 Dec 2018 13:59:26 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=EdmTrNYzqNY2Pz6uG) @RobertDiebels Thanks Robert, we have a standalone process that creates our networks soup to nuts from very simple configuration files and a few rest calls. I never extended it to automate kubectl'ing the yaml files it generates (which I did for the docker-compose version) - but using the desktop means there's only 4 steps for creation, so it isn't too bad.
Neither docker option is exactly how one would want this to work if one were to actually design this from a clean sheet - which is why I'm trying to figure out which is the best option for me. When I started this the only jira that seemed relevant was FAB-7406 which mentioned DIND. It doesn't appear that it's moved very far since I was looking at the end of summer though.
The IBM git I referenced above is far more recent, which drove my question, although given they seem to be the primary driver of this project at this time and given the paucity of opinions I think I'm leaning to sticking with what I have working until something better comes along.
sudhir.kumawat (Wed, 19 Dec 2018 12:17:07 GMT):
Hi All,
sudhir.kumawat (Wed, 19 Dec 2018 12:19:59 GMT):
I have deployed fabric 1.3 on kubernetes and accessing blockchain data using compo
sudhir.kumawat (Wed, 19 Dec 2018 12:19:59 GMT):
I have deployed fabric 1.3 on kubernetes. I have installed my business n/w over it and when i doing any transactions i am getting success but when i delete k8s nodes and bringing nodes again transaction to blockchain getting failed. I am getting some errors like
sudhir.kumawat (Wed, 19 Dec 2018 12:24:29 GMT):
Error trying invoke business network with transaction id d0c459d58988614c133ad1a94191d9eb4931f171d30df1a5c691203f00f4014e. Error: Error received from sendTransaction: Error: Failed to send transaction successfully to the orderer status:NOT_FOUND
sudhir.kumawat (Wed, 19 Dec 2018 12:25:10 GMT):
Also i am getting error in orderer like grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams failed to receive the preface from client: EOF
sudhir.kumawat (Wed, 19 Dec 2018 12:31:10 GMT):
I have tried out all cases by down peers & up again and i am getting all data same as. But i got error in only post operations to blockchain after down&up k8s nodes. Anyone can please help.
sirlarr (Thu, 20 Dec 2018 01:52:32 GMT):
Has joined the channel.
Vishal3152 (Thu, 20 Dec 2018 11:12:22 GMT):
Has joined the channel.
merq (Sat, 22 Dec 2018 03:45:14 GMT):
Has joined the channel.
ArpitKhurana1 (Wed, 26 Dec 2018 05:05:39 GMT):
@sudhir.kumawat you have to make sure you are saving the state of peer ledger and statedb
greivinlopez (Wed, 26 Dec 2018 12:57:18 GMT):
Has left the channel.
CarlosRL (Thu, 27 Dec 2018 18:04:29 GMT):
Hi, I have been running hyperledger fabric in a staging environment and I noticed that when the chaincode container remains idle for some time I get this error `14 UNAVAILABLE: TCP Write failed` with the first request. The next requests are executed just fine. I using `fabric-peer:1.2.1`. Any ideas how to address this? Thanks in advance
yousaf (Fri, 28 Dec 2018 14:23:53 GMT):
Hi everyone. I have a query that I have deployed a hyperledger fabric network on GKE instances. I want to join a peer to this fabric network so that it accesses one of these instances and becomes a part of fabric network and user can use it to perform transactions etc...??
nageshbandaru (Tue, 01 Jan 2019 20:00:29 GMT):
Has joined the channel.
xaviarias (Thu, 03 Jan 2019 11:03:07 GMT):
Has joined the channel.
x4e-salvi (Fri, 04 Jan 2019 18:48:35 GMT):
Has joined the channel.
asurirk (Tue, 08 Jan 2019 19:23:52 GMT):
Has joined the channel.
ericmvaughn (Wed, 09 Jan 2019 17:43:37 GMT):
Has joined the channel.
Skprog (Thu, 10 Jan 2019 15:23:36 GMT):
Has joined the channel.
MHBauer (Thu, 10 Jan 2019 18:46:37 GMT):
Has joined the channel.
nickgaski (Thu, 10 Jan 2019 20:35:10 GMT):
Has left the channel.
akoita (Fri, 11 Jan 2019 03:34:06 GMT):
Has joined the channel.
chinmsay213211 (Fri, 11 Jan 2019 22:23:21 GMT):
Has joined the channel.
tocosonic (Tue, 15 Jan 2019 21:40:13 GMT):
Has joined the channel.
tocosonic (Tue, 15 Jan 2019 21:41:38 GMT):
Hi *, this is a message / request to the Fabric architects:
for Hyperledger Fabric v1.1.0 I've managed to create a custom controller (openshiftcontroller) which implements the dockercontroller's interface. I've had to do 2 or 3 minor changes to the peer's code and I'm able to run Hyperledger Fabric on OpenShift in non-privileged mode :-)
Of course - I've had to create some custom images to be able to let everything run, but: it works :-)
Now my question / request:
would it be possible, to create a kind-of-a standard interface for custom controllers as well as a configuration setting by which the desired runtime environment (e.g. docker vs. openshift vs. k8s vs ...) could be selected? Right now I want to migrate from HLF 1.1.0 to HLF 1.4.0 and I have to put significant effort in finding the places where I can select the openshiftcontroller instead of the dockercontroller (and I'm hoping that the controller's interface hasn't changed that much).
What do you think?
iramiller (Wed, 16 Jan 2019 16:20:33 GMT):
@tocosonic we have a similar solution with a kubernetescontroller on our own internal fork ... I noticed there would be a lot of little changes everywhere with the protobuf definitions to properly add our Kubernetes controller if we followed the docker controller architecture currently in place... I expect that we will see such support at some point as HLF sees more production usage.
For our purposes we have minimized the effort required to support our fork by isolating our changes to HLF into our own controller and using a small shim to inject it within the standard docker controller.
```
// From core/container/dockercontroller/dockercontroller.go : line 104
// NewVM creates a new DockerVM instance
func (p *Provider) NewVM() container.VM {
// At this point check to see if we are in kubernetes
if !kubernetescontroller.InCluster() {
dockerLogger.Info("Kubernetes not detected.")
return NewDockerVM(p.PeerID, p.NetworkID, p.BuildMetrics)
}
// In a cluster so replace the docker connection with a kubernetes one.
dockerLogger.Info("Kubernetes environment detected. Using K8s API.")
return kubernetescontroller.NewKubernetesAPI(p.PeerID, p.NetworkID)
}
```
pterdchanakul (Thu, 17 Jan 2019 05:46:28 GMT):
Has joined the channel.
Yair (Fri, 18 Jan 2019 11:04:34 GMT):
Has joined the channel.
osobh (Mon, 21 Jan 2019 00:54:43 GMT):
Has joined the channel.
IanSparkes (Tue, 22 Jan 2019 08:56:04 GMT):
Has joined the channel.
DerrickL (Tue, 22 Jan 2019 15:00:11 GMT):
Has joined the channel.
incarose (Wed, 23 Jan 2019 00:23:28 GMT):
Has joined the channel.
mrjdomingus (Thu, 24 Jan 2019 10:11:01 GMT):
Has joined the channel.
Pradeep_Pentakota (Sat, 26 Jan 2019 13:42:54 GMT):
Has joined the channel.
binhn (Sat, 26 Jan 2019 16:50:21 GMT):
Has left the channel.
lip-inagora (Mon, 28 Jan 2019 00:22:21 GMT):
Has joined the channel.
JayJong (Mon, 28 Jan 2019 06:32:15 GMT):
Hi guys, my issue is that my fabric peer pods in the kubernetes cluster get evicted due to this error message: "The node was low on resource: ephemeral-storage." Im using fabric v1.3.0, kubernetes v1.13.1-00 and docker v18.06.1~ce~3-0~ubuntu. For some reason, kubernetes did bring up the evicted peer pods again but the chaincode containers are no longer in the node anymore and i have to do a join channel and install chaincode again. Does anyone knows how to fix this issue?
edisinovcic (Mon, 28 Jan 2019 13:16:02 GMT):
Has joined the channel.
frgomes (Mon, 28 Jan 2019 14:59:53 GMT):
Has joined the channel.
mattmaru (Tue, 29 Jan 2019 09:08:57 GMT):
Has joined the channel.
mattmaru (Tue, 29 Jan 2019 09:11:56 GMT):
Hello to everyone. I would get your help if possible. I need to build a fabric network using kubernetes but I could not find anything about it. Can you help me please? I'm on fabric 1.4
Hi guys. I would get your help. I must deploy an hyperledger fabric network using kubernetes but I'm not
mattmaru (Tue, 29 Jan 2019 09:12:56 GMT):
excuse me, the messages have been mixed
silliman (Tue, 29 Jan 2019 10:41:19 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=X9QEGSrF8nFKYQuXd) @mattmaru try this:
https://opensource.com/article/18/4/deploying-hyperledger-fabric-kubernetes
or this:
https://github.com/feitnomore/hyperledger-fabric-kubernetes
or you could run IBM Blockchain Platform for IBM Cloud Private community edition, which is delivered as a helm chart, on top of IBM Cloud Private community edition:
https://console.bluemix.net/docs/services/blockchain/howto/helm_install_icp.html#remote-peer-icp
https://www.ibm.com/support/knowledgecenter/en/SSBS6K_1.2.0/installing/install_containers_CE.html
IBM Cloud Private is built on top of Kubernetes. IBM Blockchain Platform is based on Hyperledger Fabric.
mattmaru (Tue, 29 Jan 2019 10:46:19 GMT):
@silliman. I have only two Ubuntu 18.10 vm and i can't use IBM
kYem 1 (Wed, 30 Jan 2019 12:57:07 GMT):
Has joined the channel.
kYem 1 (Wed, 30 Jan 2019 14:24:52 GMT):
How can i instantiate the chaincode on kubernetes peer? I'm getting timeout error, which seems to indicate the problem with booting up chaincode container?
uherr89 (Thu, 31 Jan 2019 15:49:34 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=X9QEGSrF8nFKYQuXd) @mattmaru https://worldsibu.tech/forma/
Ryan2 (Fri, 01 Feb 2019 05:14:30 GMT):
Has joined the channel.
Mozuffer (Sun, 03 Feb 2019 10:46:59 GMT):
Has joined the channel.
mattmaru (Mon, 04 Feb 2019 10:52:27 GMT):
Hello guys. I'm trying to setup kubernetes cluser (master-slave) to deploy my fabric network . Can you suggest me a guide to make it?
AndresMartinezMelgar.itcl (Mon, 04 Feb 2019 11:36:02 GMT):
Has joined the channel.
AndresMartinezMelgar.itcl (Mon, 04 Feb 2019 12:24:10 GMT):
hi, i an thinking about take kubernetes in google cloud services, but i have a simple question. That service what gives to me?? i think that is a virtual pc to admin cluster. So how can i connect from mi pc? Do i need install kubernetes into my pc??
If anyone know about a tutorial to use fabric with kubernets, i will appreciet it (with or without google cloud)
AndresMartinezMelgar.itcl (Mon, 04 Feb 2019 12:24:10 GMT):
hi, i am thinking about take kubernetes in google cloud services, but i have a simple question. That service what gives to me?? i think that is a virtual pc to admin cluster. So how can i connect from mi pc? Do i need install kubernetes into my pc??
If anyone know about a tutorial to use fabric with kubernets, i will appreciet it (with or without google cloud)
AndresMartinezMelgar.itcl (Mon, 04 Feb 2019 12:24:10 GMT):
hi, i am thinking about take kubernetes in google cloud services, but i have a simple question. That service what gives to me?? i think that is a virtual pc to admin cluster. So how can i connect from mi pc? Do i need install kubernetes into my pc??
If anyone know about a tutorial to use fabric with kubernets, i will appreciated it (with or without google cloud)
MHBauer (Mon, 04 Feb 2019 20:05:41 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=3P28RCWK6TxQ2rEGr) @AndresMartinezMelgar.itcl it's going to give you an endpoint to connect to.
MHBauer (Mon, 04 Feb 2019 20:05:48 GMT):
and you'll use kubectl
au (Wed, 06 Feb 2019 03:21:36 GMT):
Has joined the channel.
SanketPanchamia (Tue, 12 Feb 2019 03:30:31 GMT):
Has joined the channel.
AndresMartinezMelgar.itcl (Wed, 13 Feb 2019 11:06:51 GMT):
Has anyone managed to raise a network hyperledger fabric together with Kubernetes?
glennd (Wed, 13 Feb 2019 14:02:10 GMT):
Has joined the channel.
iramiller (Wed, 13 Feb 2019 22:46:52 GMT):
@AndresMartinezMelgar.itcl yes it is possible but it will require some work ... we have a multi-cloud/multi-org fabric running in production within Kubernetes.
iramiller (Wed, 13 Feb 2019 22:47:55 GMT):
Speaking of work/fabric/kubernetes ... has anyone added Istio to the mix for managing mTLS, monitoring, etc?
alexvicegrab (Sat, 16 Feb 2019 10:38:50 GMT):
https://github.com/aidtechnology/hgf-k8s-workshop
alexvicegrab (Sat, 16 Feb 2019 10:39:18 GMT):
@AndresMartinezMelgar.itcl it's a basic tutorial, but it may be a good starting point
alexvicegrab (Sat, 16 Feb 2019 10:40:12 GMT):
Working on a python library to help deploy Fabric to Kubernetes that is now in alpha, but need more hands to provide last few core features
alexvicegrab (Sat, 16 Feb 2019 10:40:26 GMT):
https://github.com/aidtechnology/nephos
alexvicegrab (Sat, 16 Feb 2019 10:40:51 GMT):
https://nephos.readthedocs.io/en/latest/py-modindex.html
AndresMartinezMelgar.itcl (Mon, 18 Feb 2019 07:01:01 GMT):
@alexvicegrab thanks, i check that lisnk right now
AndresMartinezMelgar.itcl (Mon, 18 Feb 2019 07:40:44 GMT):
My current problem is how can i keep my files in pods when these can be killed. I am trying to create a nfs server inside kubernetes cluster. I don't know if this is a right solution. Can someone guide me?
alexvicegrab (Mon, 18 Feb 2019 15:41:07 GMT):
@AndresMartinezMelgar.itcl use persistent volume claims, the links above have examples of this
CarlosRL (Mon, 18 Feb 2019 22:36:21 GMT):
Hi guys, I am using `kubernetes` to deploy fabric network in gke. It is working fine, but I want to be able to use ingress instead of services with type `LoadBalancer`. I want to have for example `peer0.org1.example.com` instead of `x.x.x.x:7051`. GRPC is supported by nginx, but only under ssl connections.
Here is an example `https://github.com/kubernetes/ingress-nginx/tree/master/docs/examples/grpc`.
The ingress requires the certificate and the key generated by `openssl`
Any ideas how to make it work
CarlosRL (Mon, 18 Feb 2019 22:36:21 GMT):
Hi guys, I am using `kubernetes` to deploy fabric network in gke. It is working fine, but I want to be able to use ingress instead of services with type `LoadBalancer`. I want to have for example `peer0.org1.example.com` instead of `x.x.x.x:7051`. GRPC is supported by nginx, but only under ssl connections.
Here is an example `https://github.com/kubernetes/ingress-nginx/tree/master/docs/examples/grpc`.
The ingress requires the certificate and the key generated by `openssl`
Any ideas how to make it work
Thanks in advance
CarlosRL (Mon, 18 Feb 2019 22:38:25 GMT):
@AndresMartinezMelgar.itcl might this help you `https://github.com/hyperledger/cello/tree/master/thirdparty/helm/fabric-chart`
CarlosRL (Mon, 18 Feb 2019 22:38:25 GMT):
@AndresMartinezMelgar.itcl might this help you `https://github.com/hyperledger/cello/tree/master/thirdparty/helm/fabric-chart`
With a helm is easier, the files can be stored in an NFS server insider the cluster.
CarlosRL (Mon, 18 Feb 2019 22:38:25 GMT):
@AndresMartinezMelgar.itcl might this help you `https://github.com/hyperledger/cello/tree/master/thirdparty/helm/fabric-chart`
With a helm is easier, the files can be stored in an NFS server inside the cluster.
ygnr (Mon, 18 Feb 2019 22:53:26 GMT):
Has joined the channel.
AndresMartinezMelgar.itcl (Tue, 19 Feb 2019 08:50:46 GMT):
thanks team, i just get it! isn't difficult when understand it
yod 15 (Wed, 20 Feb 2019 14:34:17 GMT):
Has joined the channel.
yod 15 (Wed, 20 Feb 2019 14:35:15 GMT):
@CarlosRL tryed deployment of specifyed chart and getting next errors: ```MountVolume.SetUp failed for volume "org1-cluster-shared" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/3b48060e-351a-11e9-91c7-080027a3bb5b/volumes/kubernetes.io~nfs/org1-cluster-shared --scope -- mount -t nfs -o nfsvers=4,port=30049 10.233.30.62:/cluster/resources/crypto-config/peerOrganizations/org1.example.com /var/lib/kubelet/pods/3b48060e-351a-11e9-91c7-080027a3bb5b/volumes/kubernetes.io~nfs/org1-cluster-shared Output: Running scope as unit: run-r3a8ce741364444a89edf22bc923c6be8.scope mount: wrong fs type, bad option, bad superblock on 10.233.30.62:/cluster/resources/crypto-config/peerOrganizations/org1.example.com, missing codepage or helper program, or other error (for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.
yod 15 (Wed, 20 Feb 2019 14:35:52 GMT):
However nfs pv/pvcs working correctly
AndresMartinezMelgar.itcl (Thu, 21 Feb 2019 12:23:29 GMT):
Hi again.
I have another trouvel ):
Error: failed to create deliver client: orderer client failed to connect to blockchain-orderer:31010: failed to create new connection: context deadline exceeded
AndresMartinezMelgar.itcl (Thu, 21 Feb 2019 12:30:12 GMT):
Hi again.
I have another trouvel ):
Error: failed to create deliver client: orderer client failed to connect to blockchain-orderer:31010: failed to create new connection: context deadline exceeded
i know solve this error in docker, but in kubernetes i dont. I think that is because dns-server(default cluster dns) isn't work, i tried to put ip directly but either work
I am trying with this repo https://github.com/feitnomore/hyperledger-fabric-kubernetes and i am stuck on step 12
I have a cluster with 1 node(only way to make it work) and instead of nfs-server i am using a shared folder( this folder works fine so i think this isn't error)
raj_shekhar (Thu, 21 Feb 2019 16:35:09 GMT):
Has joined the channel.
raj_shekhar (Fri, 22 Feb 2019 05:32:53 GMT):
Hi ,
I am setting a production ready fabric setup on K8s without using helm charts, I am having issues while doing configs as there are many things to configure , can anyone plz help me with it if they have done it before by sharing some repo , so that I can have much better idea?
raj_shekhar (Fri, 22 Feb 2019 12:35:49 GMT):
Hi, After installing the CA by using AID TECH tut's and Helm chart I am getting below while trying to curl CA
curl https://ca.hgf.infinichains.com/cainfo
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
raj_shekhar (Fri, 22 Feb 2019 12:36:47 GMT):
ca.hgf.infinichains.com this is my domain which is linked to my K8s cluster IP
CarlosRL (Sat, 23 Feb 2019 19:09:25 GMT):
@yod 15 In my nfs server I have all the crypto files generated and share it through some pvc volumes. For example my orderers info is in the folder `/exports/example/resources/crypto-config/ordererOrganizations/example.com`.
`/exports`: the nfs shared folder
`/exports/example/resources/crypto-config`: the crypto files generated
in my pv looks like
```apiVersion: v1
kind: PersistentVolume
metadata:
name: default-orderer-shared
spec:
accessModes:
- ReadWriteMany
claimRef:
namespace: default
name: default-orderer-shared
persistentVolumeReclaimPolicy: Retain
nfs:
path: /exports/cacao/resources/crypto-config/ordererOrganizations/monedacacao.com
server: nfs-svc.default.svc.cluster.local```
Then in the orderer deployment volumes are mounted like: (note that in the subpath the orderer is specified)
``` volumeMounts:
- mountPath: /var/hyperledger/orderer/msp
name: certificate
subPath: orderers/orderer0.monedacacao.com/msp
volumes:
- name: certificate
persistentVolumeClaim:
claimName: default-orderer-shared```
CarlosRL (Sat, 23 Feb 2019 19:09:25 GMT):
@yod15 In my nfs server I have all the crypto files generated and share it through some pvc volumes. For example my orderers info is in the folder `/exports/example/resources/crypto-config/ordererOrganizations/example.com`.
`/exports`: the nfs shared folder
`/exports/example/resources/crypto-config`: the crypto files generated
in my pv looks like
```apiVersion: v1
kind: PersistentVolume
metadata:
name: default-orderer-shared
spec:
accessModes:
- ReadWriteMany
claimRef:
namespace: default
name: default-orderer-shared
persistentVolumeReclaimPolicy: Retain
nfs:
path: /exports/cacao/resources/crypto-config/ordererOrganizations/monedacacao.com
server: nfs-svc.default.svc.cluster.local```
Then in the orderer deployment volumes are mounted like: (note that in the subpath the orderer is specified)
``` volumeMounts:
- mountPath: /var/hyperledger/orderer/msp
name: certificate
subPath: orderers/orderer0.monedacacao.com/msp
volumes:
- name: certificate
persistentVolumeClaim:
claimName: default-orderer-shared```
CarlosRL (Sat, 23 Feb 2019 19:09:25 GMT):
@yod 15 In my nfs server I have all the crypto files generated and share it through some pvc volumes. For example my orderers info is in the folder `/exports/example/resources/crypto-config/ordererOrganizations/example.com`.
`/exports`: the nfs shared folder
`/exports/example/resources/crypto-config`: the crypto files generated
in my pv looks like
```apiVersion: v1
kind: PersistentVolume
metadata:
name: default-orderer-shared
spec:
accessModes:
- ReadWriteMany
claimRef:
namespace: default
name: default-orderer-shared
persistentVolumeReclaimPolicy: Retain
nfs:
path: /exports/cacao/resources/crypto-config/ordererOrganizations/monedacacao.com
server: nfs-svc.default.svc.cluster.local```
Then in the orderer deployment volumes are mounted like: (note that in the subpath the orderer is specified)
``` volumeMounts:
- mountPath: /var/hyperledger/orderer/msp
name: certificate
subPath: orderers/orderer0.monedacacao.com/msp
volumes:
- name: certificate
persistentVolumeClaim:
claimName: default-orderer-shared```
CarlosRL (Sat, 23 Feb 2019 19:09:25 GMT):
@yod 15 In my nfs server I have all the crypto files generated and share it through some pvc volumes. For example my orderers info is in the folder `/exports/example/resources/crypto-config/ordererOrganizations/example.com`.
`/exports`: the nfs shared folder
`/exports/example/resources/crypto-config`: the crypto files generated
in my pv looks like
```apiVersion: v1
kind: PersistentVolume
metadata:
name: default-orderer-shared
spec:
accessModes:
- ReadWriteMany
claimRef:
namespace: default
name: default-orderer-shared
persistentVolumeReclaimPolicy: Retain
nfs:
path: /exports/cacao/resources/crypto-config/ordererOrganizations/monedacacao.com
server: nfs-svc.default.svc.cluster.local```
Then in the orderer deployment volumes are mounted like: (note that in the subpath the orderer is specified)
``` volumeMounts:
- mountPath: /var/hyperledger/orderer/msp
name: certificate
subPath: orderers/orderer0.example.com/msp
volumes:
- name: certificate
persistentVolumeClaim:
claimName: default-orderer-shared```
CarlosRL (Sat, 23 Feb 2019 19:09:25 GMT):
@yod 15 In my nfs server I have all the crypto files generated and share it through some pvc volumes. For example my orderers info is in the folder `/exports/example/resources/crypto-config/ordererOrganizations/example.com`.
`/exports`: the nfs shared folder
`/exports/example/resources/crypto-config`: the crypto files generated
in my pv looks like
```apiVersion: v1
kind: PersistentVolume
metadata:
name: default-orderer-shared
spec:
accessModes:
- ReadWriteMany
claimRef:
namespace: default
name: default-orderer-shared
persistentVolumeReclaimPolicy: Retain
nfs:
path: /exports/cacao/resources/crypto-config/ordererOrganizations/example.com
server: nfs-svc.default.svc.cluster.local```
Then in the orderer deployment volumes are mounted like: (note that in the subpath the orderer is specified)
``` volumeMounts:
- mountPath: /var/hyperledger/orderer/msp
name: certificate
subPath: orderers/orderer0.example.com/msp
volumes:
- name: certificate
persistentVolumeClaim:
claimName: default-orderer-shared```
CarlosRL (Sat, 23 Feb 2019 19:12:27 GMT):
@raj_shekhar can you share the logs of curl?
CarlosRL (Sat, 23 Feb 2019 19:15:30 GMT):
@AndresMartinezMelgar.itcl the deployments of your orderer are installed in the default namespace ? If not you have to specify the namespace like `blockchain-orderer.MY_NAMESPACE:31010` can you share the service of your orderer ?
raj_shekhar (Mon, 25 Feb 2019 05:00:04 GMT):
Anybody has done CA setup on K8s ??
raj_shekhar (Mon, 25 Feb 2019 06:51:30 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=zenaxg6WAQcob8SHR) @CarlosRL * TCP_NODELAY set
* Connected to ca.hgf.infinains.com (35.247....) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
HLFPOC (Mon, 25 Feb 2019 18:25:30 GMT):
Has joined the channel.
HLFPOC (Mon, 25 Feb 2019 18:29:25 GMT):
Hi Team, I have done fabric network setup using Docker swarm and now I want to do the same thing using Kubernetes. So is there any guide/reference material available which one can follow to achieve this ?
alexvicegrab (Tue, 26 Feb 2019 07:00:48 GMT):
@raj_shekhar did you correctly set up the Cert-manager?
alexvicegrab (Tue, 26 Feb 2019 07:02:09 GMT):
You know you can ping us on Github to ask about issues with CA setup, etc. So that the issues are centralised there and we can look at more promptly. Are you using the HGF workshop examples?
alexvicegrab (Tue, 26 Feb 2019 07:03:16 GMT):
@HLFPOC, yes, check our public repositories on github.com/aidtechnology, particularly Nephos, and the HGF workshop and LF webinar
HLFPOC (Tue, 26 Feb 2019 08:40:34 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=wU6jY4etjyonppjLiP) @alexvicegrab Thanks for your reply @alexvicegrab , do we have any repo which has explained the deployment without using helm charts ?
alexvicegrab (Tue, 26 Feb 2019 11:43:48 GMT):
@HLFPOC, no. Did not want to make my life more difficult :rolling_on_the_floor_laughing:
raj_shekhar (Tue, 26 Feb 2019 12:17:20 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=phh3wJGWcQtHJiMtr) right @alexvicegrab , but it will help people who are not much familiar with helm charts :wink:
raj_shekhar (Tue, 26 Feb 2019 12:17:20 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=phh3wJGWcQtHJiMtr) @alexvicegrab right @alexvicegrab , but it will help people who are not much familiar with helm charts :wink:
alexvicegrab (Tue, 26 Feb 2019 18:55:42 GMT):
Fair enough, but the point of Helm charts is not having to micromanage individual manifests. I also don't intend to make other people's life difficult in the long term
alexvicegrab (Tue, 26 Feb 2019 18:56:41 GMT):
You are free to set up your own K8S manifests without helm charts and there are examples of this elsewhere, but it's not my road to walk
alexvicegrab (Tue, 26 Feb 2019 18:58:13 GMT):
A helm chart is ultimately a grouping of Kubernetes manifests with the possibility of using a very simple templating language. And it's why we made the workshops and Webinar. If anything is unclear do leave comments on the relevant repos we created. We tried to document as well as we could but doubt we have been perfect in everything
alexvicegrab (Tue, 26 Feb 2019 19:02:49 GMT):
First video of workshop is here in case this helps https://youtu.be/ubrA3W1JMk0
ygnr (Tue, 26 Feb 2019 22:47:31 GMT):
@alexvicegrab Nice job on helm charts! I see that your are mounting docker.sock from node. How are you maintaining chaincode upgrades?
ygnr (Tue, 26 Feb 2019 22:48:13 GMT):
Any reasons why you didn't consider docker in docker?
raj_shekhar (Wed, 27 Feb 2019 05:13:52 GMT):
@alexvicegrab yeah ,, perfectly said..
AndresMartinezMelgar.itcl (Wed, 27 Feb 2019 07:58:17 GMT):
hi, someone have a nfs-server within kubernetes? can guide me?
I need it to share file among diferent nodes
alexvicegrab (Wed, 27 Feb 2019 12:44:35 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=mzySDACJHkmzPNfd5) @ygnr For our POCs, we've used Composer, but are now working on shifting to raw Fabric. We are working on making chaincode deployment/updates easier as we speak.
AndresMartinezMelgar.itcl (Thu, 28 Feb 2019 12:29:01 GMT):
Hi, i have a nfs-server out of kubernetes but i get that network work rightly
I want to create nfs-server within cluster, do anyone prove it?
I readed nfs-server must be external to kubernetes's cluster
Estebanrestrepo (Fri, 01 Mar 2019 19:39:41 GMT):
Has joined the channel.
SaicharanPogul (Sun, 03 Mar 2019 13:47:38 GMT):
Has joined the channel.
AndresMartinezMelgar.itcl (Mon, 04 Mar 2019 12:11:20 GMT):
hi. I already have a cluster with a functional hyperledger fabric. So how can I send transactions from outside the cluster?
Can it be valid with curl or something similar?
AndresMartinezMelgar.itcl (Mon, 04 Mar 2019 12:11:20 GMT):
hi. I already have a cluster with a functional hyperledger fabric(orderer has mode solo, i dont get kafka yet). So how can I send transactions from outside the cluster?
Can it be valid with curl or something similar?
AndresMartinezMelgar.itcl (Mon, 04 Mar 2019 12:11:20 GMT):
hi. I already have a cluster with a functional hyperledger fabric(orderer has mode solo, i havent get kafka yet). So how can I send transactions from outside the cluster?
Can it be valid with curl or something similar?
aatkddny (Mon, 04 Mar 2019 18:22:10 GMT):
look at a nodeport
aatkddny (Mon, 04 Mar 2019 18:22:10 GMT):
look at a nodeport if you want a simple setup
aatkddny (Mon, 04 Mar 2019 18:23:43 GMT):
```
apiVersion: v1
kind: Service
metadata:
name: peer0-myorg
spec:
type: NodePort
ports:
- name: listen-endpoint
protocol: TCP
nodePort: 30201
port: 7051
targetPort: 7051
- name: chaincode-endpoint
protocol: TCP
nodePort: 30202
port: 7052
targetPort: 7052
- name: event-endpoint
protocol: TCP
nodePort: 30203
port: 7053
targetPort: 7053
selector:
app: peer0-myorg
```
You'd send grpc to port 30201 for peer0 of myorg.
aatkddny (Mon, 04 Mar 2019 18:23:43 GMT):
```
apiVersion: v1
kind: Service
metadata:
name: peer0-myorg
spec:
type: NodePort
ports:
- name: listen-endpoint
protocol: TCP
nodePort: 30201
port: 7051
targetPort: 7051
- name: chaincode-endpoint
protocol: TCP
nodePort: 30202
port: 7052
targetPort: 7052
- name: event-endpoint
protocol: TCP
nodePort: 30203
port: 7053
targetPort: 7053
selector:
app: peer0-myorg
```
You'd send grpc to port 30201 for peer0 of myorg. This one (obviously) is for 1.1/1.2 - later versions remove the event stuff.
braduf (Tue, 05 Mar 2019 00:05:52 GMT):
Has joined the channel.
aatkddny (Tue, 05 Mar 2019 16:33:16 GMT):
Anyone manage to get 1.4.0 with gossip working?
No matter what I try I'm getting:
`2019-03-05 15:51:05.059 UTC [gossip.discovery] func1 -> WARN 035 Could not connect to Endpoint: peer1-myorg:30061, InternalEndpoint: peer0-myorg:30061, PKI-ID:
aatkddny (Tue, 05 Mar 2019 16:33:16 GMT):
Anyone manage to get 1.4.0 with gossip working?
No matter what I try I'm getting:
`2019-03-05 15:51:05.059 UTC [gossip.discovery] func1 -> WARN 035 Could not connect to Endpoint: peer1-myorg:30061, InternalEndpoint: peer0-myorg:30061, PKI-ID:
aatkddny (Tue, 05 Mar 2019 16:33:16 GMT):
Anyone manage to get 1.4.0 with gossip working?
No matter what I try I'm getting:
`2019-03-05 15:51:05.059 UTC [gossip.discovery] func1 -> WARN 035 Could not connect to Endpoint: peer1-myorg:30061, InternalEndpoint: peer0-myorg:30051, PKI-ID:
yacovm (Tue, 05 Mar 2019 18:03:47 GMT):
@aatkddny try to set the gRPC logger maybe you have a TLS problem
yacovm (Tue, 05 Mar 2019 18:05:22 GMT):
`FABRIC_LOGGING_SPEC=grpc=debug`
yacovm (Tue, 05 Mar 2019 18:05:37 GMT):
https://hyperledger-fabric.readthedocs.io/en/release-1.4/logging-control.html
aatkddny (Tue, 05 Mar 2019 18:18:07 GMT):
@yacovm - TLS isn't on. I can connect to the peer from outside my cluster - this is inside our DC so I haven't bothered with an ingress - it's all NodePorts.
aatkddny (Tue, 05 Mar 2019 18:18:07 GMT):
@yacovm - TLS isn't on. I can connect to the peer from outside my cluster - this is inside our DC so I haven't bothered with an ingress or any TLS - it's all NodePorts.
aatkddny (Tue, 05 Mar 2019 18:18:07 GMT):
@yacovm - TLS isn't on. I can connect to the peer from outside my cluster - this is inside our DC so I haven't bothered with an ingress or any TLS - it's all NodePorts and gRPC.
yacovm (Tue, 05 Mar 2019 18:18:42 GMT):
I hope you activate TLS regardless where this runs
yacovm (Tue, 05 Mar 2019 18:19:00 GMT):
is the entire blockchain running in your DC?
aatkddny (Tue, 05 Mar 2019 18:19:04 GMT):
Yes.
aatkddny (Tue, 05 Mar 2019 18:19:12 GMT):
It's all running in the same cluster.
aatkddny (Tue, 05 Mar 2019 18:19:17 GMT):
This is a test setup.
yacovm (Tue, 05 Mar 2019 18:19:22 GMT):
then why do you need a blockchain? use a distributed database
yacovm (Tue, 05 Mar 2019 18:19:25 GMT):
ah, test setup... ok
yacovm (Tue, 05 Mar 2019 18:20:04 GMT):
ok try to do a tcpdump recording and open with wireshark
yacovm (Tue, 05 Mar 2019 18:20:17 GMT):
and see what is going on - i.e, is it a name resolution problem perhaps
yacovm (Tue, 05 Mar 2019 18:20:22 GMT):
or is there no connectivity
yacovm (Tue, 05 Mar 2019 18:20:30 GMT):
do you receive an RST from the remote host
yacovm (Tue, 05 Mar 2019 18:20:32 GMT):
or nothing
yacovm (Tue, 05 Mar 2019 18:20:34 GMT):
etc. etc.
aatkddny (Tue, 05 Mar 2019 18:21:33 GMT):
Ugh - I was hoping it was an obvious misconfiguration. Off to debug hell I go...
yacovm (Tue, 05 Mar 2019 18:21:52 GMT):
why debug hell?
yacovm (Tue, 05 Mar 2019 18:21:55 GMT):
debug paradise
aatkddny (Tue, 05 Mar 2019 18:23:42 GMT):
Because now I have to take time out from what I wanted to do to figure out what's going wrong. I have my own stuff that needs debugging too. :D
aatkddny (Tue, 05 Mar 2019 19:37:38 GMT):
What does `network ID` have to do with this?
aatkddny (Tue, 05 Mar 2019 19:45:08 GMT):
OK to close this off. It doesn't find peer1-myorg:30061 despite the NodePort service being extant.
It does work with peer1-myorg:7051 though, so I'll just leave it there and pretend this never happened...
mauricio (Wed, 06 Mar 2019 21:42:11 GMT):
Has joined the channel.
raylau12 (Thu, 07 Mar 2019 01:53:07 GMT):
Has joined the channel.
raylau12 (Thu, 07 Mar 2019 02:10:44 GMT):
hi, I am facing a problem that I would like to startup the openshift console after the installation, an "The connection to the server [host] was refused - did you specify the right host or port?" error occurred in my master VM
raylau12 (Thu, 07 Mar 2019 02:10:50 GMT):
does anyone can help?
mamtabhardwaj12 (Fri, 08 Mar 2019 10:12:36 GMT):
Hey, Is there anyone who have done IBM Blockchain platform on IBM cloud private?
AndresMartinezMelgar.itcl (Fri, 08 Mar 2019 16:46:11 GMT):
i have deploy a new on gcp
JorgeNavarro (Mon, 11 Mar 2019 12:06:15 GMT):
Has joined the channel.
KyunghoKim (Tue, 12 Mar 2019 03:09:23 GMT):
Has joined the channel.
raj_shekhar (Tue, 12 Mar 2019 06:55:29 GMT):
Need some quick info.....
what is the diff between "configMap" and "configMapRef" tags in K8s
iamdm (Wed, 13 Mar 2019 09:54:12 GMT):
Has joined the channel.
krabradosty (Wed, 13 Mar 2019 21:50:21 GMT):
Hi folks! Could anyone give own thoughts or provide link: what are the advantages of deploying Fabric with kubernetes? I see so many problems people face, but I don't understand why they are choosing kubernetes. Thanks for any help!
raj_shekhar (Thu, 14 Mar 2019 11:02:54 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=RELnXsizbbvqQqvqe) @krabradosty I am also exploring this particularly, the main benefit I think is the better container/resource management and release management ,,,,
raj_shekhar (Thu, 14 Mar 2019 11:03:20 GMT):
Let me know what issues others are facing in it...
krabradosty (Thu, 14 Mar 2019 11:13:46 GMT):
@raj_shekhar Mainly these issues are caused by a bad understanding of kubernetes. It is not a problem of Fabric.
raj_shekhar (Thu, 14 Mar 2019 11:20:43 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=Fp9fmt74BAkKciq4q) @krabradosty right
iramiller (Thu, 14 Mar 2019 14:31:59 GMT):
@krabradosty deploying a production system based on containers means that orchestration is a required line item for any non-trivial system. While the learning curve for Kubernetes is steep given the number of containers required for even a single Org fabric system there really isn't a better solution out there—especially if you are going into a public cloud. Our company has a fabric 1.4 deployment with several organizations spread across GKE and AWS (soon to include Azure). At this scale there really isn't any other way. Unfortunately in 1.4 the Fabric chaincode scheduling is not sufficient for our needs which required us to implement Kubernetes native scheduling of pods instead of using the docker.sock approach. It appears that Fabric 2.0 has a major goal of streamlining this area (https://jira.hyperledger.org/browse/FAB-13582).
iramiller (Thu, 14 Mar 2019 14:38:07 GMT):
The impression [assumption] I have of your familiarity with Kubernetes and container orchestration for production systems based on this question leads me to believe I should warn you that acquiring proficiency in kubernetes is an extremely non-trivial undertaking which will require a great deal of time to achieve. Further there are many places where getting it wrong (ingress, volumes, security) could result in grave risks to your deployments. That said it is a very worthwhile skill to acquire for any systems deployment in the cloud.
krabradosty (Thu, 14 Mar 2019 15:12:18 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=77ENDFkJ4DamqWeJN) @iramiller Could you please provide some real use-cases of orchestration you mentioned here? I understand the advantages of using kubernetes with stateless applications. But Fabric nodes are stateful. Moreover, to run for example a new peer, some prerequisites have to be done (registering a peer identity in CA, ...). So as I see it, Fabric will not work with kubernetes out of the box. And my point is maybe it would be easier to implement own orchestration tool for Fabric network (based on ansible for example) rather than trying to "integrate" stateful and complex Fabric with k8s?
Also, I'm curious, do you deploy your databases, message brokers, etc on k8s cluster? I googled this question a lot, but can't find a strict answer with pros and cons. Only answers like "you can if you want".
And thanks for the warning :grinning:
iramiller (Thu, 14 Mar 2019 15:37:17 GMT):
@krabradosty well... 'out of the box' is a somewhat vague concept here given the extensive set of tools available and the variety of use cases. I would consider our environment to be very close to the Fabric software as released. We added about 100-120 lines of Golang to our fork for kubernetes integration in the peer process itself but everything else is based on released binaries. We do have extensive systems tooling in TerraForm, Ansible, and good old bash scripts to make our environment manageable.
Orchestration uses that are most valuable to us are the ability to define the resource requirements of the various components of our system and rely on Kubernetes to correctly schedule these components across the available nodes. All of our peer deployments run as StatefulSets which Kubernetes handles in a more appropriate way for scheduling. For databases we prefer to use solutions like Google's Cloud SQL outside of Kubernetes although for certain situations we do run small databases (postgres, Redis, etc) as containers within Kubernetes.
In a public cloud the machines these systems are deployed on WILL be restarted/fail/migrate with downtime. A solution that pretends the machine will be up consistently is going to have a bad time. As a concrete example there was a recent CVE issued which required all of our nodes to be rebooted for patching. This was handled without impact to our production workflows (other than some additional retries occurring for some transactions). We also run a very large Kafka cluster within Kubernetes with good results. The Kafka/messaging architecture has been crucial to our success as a Request/Reply architecture is not very compatible with a distributed system.
It is better to jump in from the start believing that pieces can and will fail unexpectedly. For reference our scale is around a dozen fabric orgs each with multiple peers and an ICA. We have instances in every AWS availability zone across North America and many GKE zones as well.
iramiller (Thu, 14 Mar 2019 15:57:16 GMT):
One more important point... Our scale and environment is driven by business requirements related to our 'distributed trust problem' which the blockchain approach solves and not be technical requirements for this size environment. We absolutely do not need all of this hardware for throughput nor would we scale in this way strictly for availability.
krabradosty (Thu, 14 Mar 2019 16:14:14 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=Tkity2ZXND64KBuj6) @iramiller You mean that with Kubernetes we can achieve required availability without high redundancy of hardware?
iramiller (Thu, 14 Mar 2019 16:22:47 GMT):
@krabradosty that really isn't the take away I was going for here. My point was that I wouldn't choose to have 30-50 machines in this environment spread across the continent solely for the sake of availability or capacity. Kubernetes scheduling does let us minimize the size of hardware we use, supports the use of preemptive nodes on GKE which are much cheaper, and scale out more hardware seamlessly as load dictates.
krabradosty (Thu, 14 Mar 2019 16:42:13 GMT):
Many thanks for your detailed response
raj_shekhar (Thu, 14 Mar 2019 16:47:14 GMT):
@iramiller great information ,,, Thanks a lot
yacovm (Thu, 14 Mar 2019 16:48:31 GMT):
@iramiller do you run kafka in a single organization or in several ?
yacovm (Thu, 14 Mar 2019 16:48:47 GMT):
or - do you run it in diff. avail. zones?
iramiller (Thu, 14 Mar 2019 16:49:46 GMT):
Our consensus Org is as single dedicated centralized 5 node Kafka/ZK cluster ... we have a separate cluster for our application layer ... and member orgs of course can do their own thing as well.
yacovm (Thu, 14 Mar 2019 16:50:05 GMT):
i see. so the consensus is still centralized
iramiller (Thu, 14 Mar 2019 16:50:14 GMT):
@yacovm we are looking forward to Raft...
yacovm (Thu, 14 Mar 2019 16:50:28 GMT):
it's almost out the door... in its final testing stages
iramiller (Thu, 14 Mar 2019 16:51:54 GMT):
Additionally we believe internally that we may need to migration our system to a service mesh (think Istio, similar) to streamline various aspects. Currently we use a Wireguard and dedicated namespace architecture I designed which allows all of the distributed clusters to appear to be a single uber cluster.
yacovm (Thu, 14 Mar 2019 16:52:00 GMT):
you can take a sneak peak at a high level administration document [here](https://logs.hyperledger.org/production/vex-yul-hyp-jenkins-3/fabric-docs-build-x86_64/1445/html/raft_configuration.html)
yacovm (Thu, 14 Mar 2019 16:52:20 GMT):
if you migrate to a service mesh... beware that Raft OSNs use mutual TLS for authentication
iramiller (Thu, 14 Mar 2019 16:52:57 GMT):
The service mesh migration is further influenced by the proposed 2.0 changes to the chaincode/peer relationship proposed in HLF 2.0
yacovm (Thu, 14 Mar 2019 16:53:00 GMT):
and TLS pinning
iramiller (Thu, 14 Mar 2019 16:53:20 GMT):
We use mutual TLS everywhere today.
yacovm (Thu, 14 Mar 2019 16:53:36 GMT):
> The service mesh migration is further influenced by the proposed 2.0 changes to the chaincode/peer relationship proposed in HLF 2.0
that's a very polite way of saying that the current peer<---> docker is not cloud friendly
yacovm (Thu, 14 Mar 2019 16:53:50 GMT):
you might use mutual TLS but you should beware of TLS terminating proxies
yacovm (Thu, 14 Mar 2019 16:54:04 GMT):
you're going to need TLS passthrough proxies
yacovm (Thu, 14 Mar 2019 16:54:25 GMT):
because when node A contacts node B, node B infers its ID from the TLS client certificate of node A
yacovm (Thu, 14 Mar 2019 16:54:39 GMT):
just a heads up, of course
iramiller (Thu, 14 Mar 2019 16:54:47 GMT):
@yacovm we rewrote the peer code to use a kubernetescontroller injected to replace the docker controller which makes our deployments cloud friendly
yacovm (Thu, 14 Mar 2019 16:55:46 GMT):
I see. well, if you'll need some customization for Raft feel free to ping me and I'll either try to add it to the official Raft code or can help you fork it
yacovm (Thu, 14 Mar 2019 16:55:46 GMT):
I see. well, if you'll need some customization for Raft feel free to ping me and I'll either try to add it to the official Raft (if it makes sense) code or can help you fork it
iramiller (Thu, 14 Mar 2019 16:56:23 GMT):
further in our environment any member node can access services in by dns name of `peer-0.membername.svc.cluster.local` ... which is on the TLS certificate (or the simplified `peer-0.membername`)
iramiller (Thu, 14 Mar 2019 16:57:25 GMT):
We were just discussing this morning that we should internally gather a 'wishlist' of sorts to discuss further with the HLF team as a way which we might be able to better engage with the community and potentially make contributions.
yacovm (Thu, 14 Mar 2019 16:57:57 GMT):
yes we welcome external contributions
yacovm (Thu, 14 Mar 2019 16:58:03 GMT):
sadly most of them are typo fixes
iramiller (Thu, 14 Mar 2019 16:58:09 GMT):
Contributing back to the community is a stated high level goal fo the company ... the details for engagement and release of information ... well that is more complicated and outside of my lane unfortunately
iramiller (Thu, 14 Mar 2019 16:58:09 GMT):
Contributing back to the community is a stated high level goal of the company ... the details for engagement and release of information ... well that is more complicated and outside of my lane unfortunately
iramiller (Thu, 14 Mar 2019 16:59:49 GMT):
Internally for reference the release of our kubernetescontroller was hung up due to other priorities of which as a startup are many and all high ... from engineerings perspective their is concern that as our solution does not fully integrate or seamless pass these tests it would not be a useful contribution.
iramiller (Thu, 14 Mar 2019 16:59:49 GMT):
Internally for reference the release of our kubernetescontroller was hung up due to other priorities of which as a startup are many and all high ... from engineerings perspective their concern is that as our solution does not fully integrate or seamless pass these tests it would not be a useful contribution.
iramiller (Thu, 14 Mar 2019 17:10:07 GMT):
```
All TLS certificates have an expiration date that is determined by the issuer. These expiration dates can range from 10 years from the date of issuance to as little as a few months, so check with your issuer. Before the expiration date, you will need to rotate these certificates on the node itself and every channel the node is joined to, including the system channel.```
The TLS management and rotation are one of the major potential benefits we expect to see when migrating to a service mesh. Our infosec group would like to see the rotation time driven down significantly to ensure the process runs smooth and is well managed. Given the operational complexity of running a hyper ledger fabric production environment will increase with Raft and the future Peer/Chaincode refactor it would seem that adopting a recommended deployment pattern that is tested and supported maybe in the best interest of the project. My personal preference is obviously for a kubernetes driven solution such as Istio however I also am acutely aware that such a solution is incredibly daunting for a small team without resources that are familiar with all of these external components.
iramiller (Thu, 14 Mar 2019 17:49:43 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=9LrxWNDXptGaBFQa4) To clarify on the peer scheduling requiring registration with the channels, CA, etc. We do not scale the number of peers up/down in kubernetes. We schedule the peers on nodes relying on the resource scheduling capabilities of Kubernetes. To support management of secrets and configuration we store all of the config and certificates in configmaps and secrets within kubernetes. We do use a PVC for the chain data. Further we use multiple containers within the same pod, one for the peer and the second for couchdb which means that the couch connection is always localhost to the peer for performance. These are some of the useful aspects of Kubernetes orchestration even for stateful workloads like a peer process. Additionally we schedule our chaincode as one off pods managed by the peer itself via Kubernetes API calls and rely on NetworkPolicy configuration to ensure the scheduled peer chaincode containers can only talk to their owning peer container.
yacovm (Thu, 14 Mar 2019 17:55:48 GMT):
@iramiller why do you need a service mesh for fabric? I don't understand... can you enlighten me please?
yacovm (Thu, 14 Mar 2019 17:56:07 GMT):
Fabric manages security and load balancing and failover on its own
iramiller (Thu, 14 Mar 2019 19:41:56 GMT):
@yacovm the areas we are exploring a service mesh for to solve are 1. -- TLS management (a quick scan shows nearly 600 instances of pem files for MSPs, TLS, mutual TLS identities in our environment). 2. -- Secure cross cluster connections, we may be able to integrate our Wireguard VPN system into a service mesh for more automated rollout and administration. 3. -- Service routing solution for the updated chaincode/peer architecture proposed for HLF 2.0. We require control of the versions of chaincode used within our consortium and we must have multiple versions deployable and a single version instantiated across the entire network. Using a service mesh we can handle routing between these version. Additionally we use a custom end-to-end encryption system that encrypts the entire contents of every transaction. This approach uses customer managed keys so the platform itself does not have access to the data. As part of this approach the client that submits a transaction determines which of the many peers available it wants to use and includes an ephemeral public key from the chaincode instance allowed to process the transaction. I am simplifying this a little bit but the short of all of that is that we have our own service and instance discover system with key management and we believe that a service mesh could potentially be used to streamline this process further.
iramiller (Thu, 14 Mar 2019 19:43:32 GMT):
Additionally I would like to point out that the adoption and potential migration to a service mesh is in the research, scoping, and prototyping phase. We have not committed to this direction although there are a few initial promising aspects associated with it.
yacovm (Thu, 14 Mar 2019 19:44:24 GMT):
i don't see how application level encrypted is remotely related to service mesh
yacovm (Thu, 14 Mar 2019 19:45:25 GMT):
as for VPN - I understand why someone would want to use a VPN
iramiller (Thu, 14 Mar 2019 19:45:27 GMT):
service discovery and registration/rotation of the ephemeral public key for the chaincode endpoint is the overlap we have annotated
iramiller (Thu, 14 Mar 2019 19:45:48 GMT):
I listed it last as it is the least of the priorities
yacovm (Thu, 14 Mar 2019 19:45:48 GMT):
are the peers also running behind istio?
yacovm (Thu, 14 Mar 2019 19:46:02 GMT):
because it might interfere with fabric's native service discovery
yacovm (Thu, 14 Mar 2019 19:46:07 GMT):
unless you don't use it
iramiller (Thu, 14 Mar 2019 19:46:36 GMT):
we would potentially run upto 100% of the inter container connectivity through Istio ... and no ... we are not currently using the service discovery offering in Fabric
yacovm (Thu, 14 Mar 2019 19:46:49 GMT):
why not?
iramiller (Thu, 14 Mar 2019 19:50:29 GMT):
We already needed to have a system that captures a generated public key from the chaincode container instance within a member. By design this key is maintained in memory and rotated frequently as it is associated with transaction processing and is the single point within our environment when a customers transaction is decrypted. We have integrity monitoring controls wrapping all of these instances to further protect this environment with a forward looking design intended to work with a TEE when that matures. As part of all of this our centralized key management server maintains a definitive list of authorized chaincode containers within the environment (subject to revocation based on security controls and policy). No transactions can be submitted to our network and processed by chaincode without contacting this kms. As such we did not see any benefit in adopting the discovery service (at this time).
yacovm (Thu, 14 Mar 2019 19:53:52 GMT):
what fabric's service discovery is, is a service that the client (SDK) asks it "which peers should I ask endorsement from for chaincode *foo*?" and it returns an answer based on the endorsement policy
yacovm (Thu, 14 Mar 2019 19:54:30 GMT):
I don't really understand how your TEE and encryption use case is making it useless
iramiller (Thu, 14 Mar 2019 19:54:49 GMT):
that is consistent with our understanding of it as well...
iramiller (Thu, 14 Mar 2019 19:55:06 GMT):
it is not useless... only largely redundant in our environment ...
yacovm (Thu, 14 Mar 2019 19:55:26 GMT):
ok so when you change the endorsement policy do you rewrite the application?
yacovm (Thu, 14 Mar 2019 19:55:39 GMT):
and when you add new peers do you have the application query k8s or something?
iramiller (Thu, 14 Mar 2019 19:55:40 GMT):
why would we do that?
iramiller (Thu, 14 Mar 2019 19:56:01 GMT):
Our KMS maintains information on every active peer and chaincode instance across the consortium
iramiller (Thu, 14 Mar 2019 19:56:26 GMT):
further is is aware of the current status of security policy with regards to which instances are authorized
yacovm (Thu, 14 Mar 2019 19:56:28 GMT):
let's say you have chaincode *foo* and it has endorsement policy *org1* and *org2* and now you add a new org to the channel - *org3*.
and you upgrade the chaincode and change the endorsement policy to be *org1* and *org2* and *org3*
yacovm (Thu, 14 Mar 2019 19:56:34 GMT):
how does you application adapt ?
iramiller (Thu, 14 Mar 2019 19:59:32 GMT):
Our KMS is aware of the endorsement policy ... it is queried for a set of potential peer/chaincode instances for the associated operation ... the results of this query include the public keys for those chaincode instances... the client can make its own determination of which of the members to use or not use based on their own internal policies for which orgs they wish to have potential exposure to their information (however brief). The client the encrypts the transaction and includes a collection of keys encrypted with the public keys of audience members allowed to decrypt the payload (which includes peers, other identities, etc)
yacovm (Thu, 14 Mar 2019 20:00:15 GMT):
ok i see
iramiller (Thu, 14 Mar 2019 20:00:20 GMT):
our KMS concept was in place before we migrated from 1.1 ...
yacovm (Thu, 14 Mar 2019 20:00:31 GMT):
your company is light years ahead than 99% of fabric deployments I would presume
iramiller (Thu, 14 Mar 2019 20:01:15 GMT):
We do have a large investment in blockchain and a vested interest in ensuring the continued success of HLF
iramiller (Thu, 14 Mar 2019 21:00:21 GMT):
@yacovm
```
As we noted in the conceptual documentation, leader elections in Raft are triggered when follower nodes do not receive either a “heartbeat” messages or an “append” message that carries data from the leader for a certain amount of time. ```
^^^ monitoring network performance with Istio would be pretty useful for this reason -- https://github.com/GoogleCloudPlatform/istio-samples/tree/master/istio-stackdriver
yacovm (Thu, 14 Mar 2019 21:28:44 GMT):
how can you do that @iramiller ?
yacovm (Thu, 14 Mar 2019 21:29:05 GMT):
Raft uses TLS
yacovm (Thu, 14 Mar 2019 21:29:18 GMT):
and fabric uses ephemeral diffie helman key exchange
yacovm (Thu, 14 Mar 2019 21:29:32 GMT):
so you are not supposed to be able to see what is going on in the connection
iramiller (Thu, 14 Mar 2019 21:50:02 GMT):
well... the TLS is managed by an Istio sidecar container... Istio understands all of the routes in the network and captures metrics and instrumentation on them.
iramiller (Thu, 14 Mar 2019 21:50:46 GMT):
essentially if migrating to a service mesh the ideal situation is a separation of concerns between the network and transport related aspects of a program and its endpoints/internal logic
iramiller (Thu, 14 Mar 2019 21:52:01 GMT):
As one can quickly imagine migrating to such a setup is non-trivial and potentially an invasive change to the software architecture.
iramiller (Thu, 14 Mar 2019 21:53:10 GMT):
And yet if this is done all of the TLS related concerns fall away as they are abstracted out (minus some sort of bridge which would be required to memorialize the official current configuration in the blockchain that reflects the known identities.
iramiller (Thu, 14 Mar 2019 22:00:55 GMT):
For those following along it maybe helpful to read the overview of Istio security: https://istio.io/docs/concepts/security/ [covers mutual tls, etc] and further contemplate the benefits of separating the traffic management concerns out of fabric by reviewing https://istio.io/docs/concepts/traffic-management/
iramiller (Thu, 14 Mar 2019 22:02:13 GMT):
These capabilities are the true power of a cloud orchestration platform and are really a next level solution. This is likely more that almost all users of HLF are looking for... but if you have a production system in the cloud of any significant size then it is absolutely where you want to be
yacovm (Fri, 15 Mar 2019 00:11:16 GMT):
> well... the TLS is managed by an Istio sidecar container...
but you can't run Raft orderers without TLS. It is just impossible code wise
iramiller (Fri, 15 Mar 2019 15:34:14 GMT):
@yacovm the TLS certificates are generated and managed by Istio and saved as secrets in Kubernetes which are then mounted into the container... the HLF processes will not see any difference from any other pem file when loading their secret.
aatkddny (Sun, 17 Mar 2019 22:47:03 GMT):
Cross post from the #fabric-sdk-java channel. I'm trying to switch from NodePort to Ingress on my test setup.
The first one I tried was the CA - so I mapped localhost/ca to the correct service, added a little rewrite code and it works fine.
Until I try to do it from the sdk - which requires urls of the format host:port.
Kubernetes isn't my strongest subject, so I'm wondering if there's a workaround here that'll let me get to these - and the peers and orderers - without the port count restrictions of the NodePort, and with the handy SSL/GRPCS features I can get from the Ingress.
aatkddny (Sun, 17 Mar 2019 22:47:03 GMT):
Cross post from the #fabric-sdk-java channel. I'm trying to switch from NodePort to Ingress on my test setup.
The first one I tried was the CA - so I mapped localhost/ca to the correct service, added a little rewrite code and it works fine in Postman.
Sadly it's less wonderful when I try to do it from the sdk - which requires urls of the format host:port.
Kubernetes isn't my strongest subject, so I'm wondering if there's a workaround here that'll let me get to these - and the peers and orderers - without the port count restrictions of the NodePort, and with the handy SSL/GRPCS features I can get from the Ingress.
LeeJCherry (Mon, 18 Mar 2019 21:48:36 GMT):
Has joined the channel.
LeeJCherry (Mon, 18 Mar 2019 21:49:29 GMT):
Any K8s on AWS guides out there? New to k8s and HLF. Making for an interesting few days!
iramiller (Mon, 18 Mar 2019 23:36:16 GMT):
In my experience AKS is not quite as nice as GKE ... but the deployments are quite similar if you are using nodeports for your services.
iramiller (Mon, 18 Mar 2019 23:37:51 GMT):
@LeeJCherry if you find a guide for kubernetes installs of fabric on any platform it will probably be worth a look even if they are not targeting AWS/AKS specifically
alexvicegrab (Tue, 19 Mar 2019 00:34:45 GMT):
@LeeJCherry, you can also try https://github.com/hyperledger-labs/nephos
alexvicegrab (Tue, 19 Mar 2019 00:34:45 GMT):
@LeeJCherry, you can also try https://github.com/hyperledger-labs/nephos
It should work fine for deploying Fabric on AWS & GCP (tested on Azure and Minikube)
alexvicegrab (Tue, 19 Mar 2019 00:36:23 GMT):
One of the (many) good points of Kubernetes is to make deployments cloud-agnostic
AndresMartinezMelgar.itcl (Tue, 19 Mar 2019 09:03:23 GMT):
Hi, someone receive this error in k8 ? Error: could not assemble transaction, err proposal response was not successful, error code 500, msg cannot get package for chaincode(cc:1.0)
LeeJCherry (Tue, 19 Mar 2019 17:40:46 GMT):
Thanks @iramiller.
LeeJCherry (Tue, 19 Mar 2019 17:42:51 GMT):
@alexvicegrab I've been following one of the other aid tech Github repos readme and a you tube video they have. Getting all sorts of errors which I think is due to the guide being a little out of date. There is mention of the Nephos repo as well. What's the difference? I'm not using EKS, I'm going for k8s purely for the cloud agnostic benefit
LeeJCherry (Tue, 19 Mar 2019 17:43:39 GMT):
@alexvicegrab actually just noticed! It's your video I have been watching!!
alexvicegrab (Tue, 19 Mar 2019 17:58:57 GMT):
@LeeJCherry, the GitHub webinar and workshop are a bit older, but should work. Yes, they are my videos. Nephos basically automates the tests. We have unit tests, integration tests, documentation, code coverage and code quality monitoring with SonarQube and security auditing with Snyk. It *should* work, even though it's a late alpha or early beta
LeeJCherry (Tue, 19 Mar 2019 18:28:58 GMT):
Thanks @alexvicegrab I'll give it a spin!
diegoaosuna (Tue, 19 Mar 2019 19:26:18 GMT):
Has joined the channel.
sunlidong (Wed, 20 Mar 2019 10:12:49 GMT):
Has joined the channel.
yousaf (Fri, 22 Mar 2019 10:10:11 GMT):
Hi everyone. I am deploying HL fabric network on kubernetes & facing the error below on joining peers with the channel.
For org1peer0 and org2peer0, joining is successful. But for org1peer1 and org2peer1 joining gives the error below.
Error: error getting endorser client for channel: endorser client failed to connect to blockchain-org1peer1:30112: failed to create new connection: context deadline exceeded
Any suggestions or fixes about the issue above?
AbhijeetRastogi (Fri, 22 Mar 2019 18:20:01 GMT):
Has joined the channel.
yanli133 (Mon, 25 Mar 2019 06:54:48 GMT):
Has joined the channel.
aatkddny (Tue, 26 Mar 2019 23:40:53 GMT):
anyone get docker in docker running without it being in a sidecar container? meaning as a single standalone but still able to be called by the peers? i've wasted most of a day trying to get this to work and not got as close as i'd like to have.
aatkddny (Tue, 26 Mar 2019 23:40:53 GMT):
anyone get docker in docker running without it being in a sidecar container? meaning as a single standalone but still able to be called by the peers? i've wasted most of a day trying to get this to work and not got as close as i'd like to have.
VictorStroganov (Wed, 27 Mar 2019 09:02:46 GMT):
Hi all! I have HLF running on Kubernetes and got a problem during nodejs chaincode instantiation:
fabric-ccenv container hangs with "npm sill install loadAllDepsIntoIdealTree" message.
On the host "npm install" for the same chaincode works good, but it hangs in container.
Can anyone help?
Kyroy (Thu, 28 Mar 2019 13:40:42 GMT):
Has left the channel.
raj_shekhar (Fri, 29 Mar 2019 04:40:00 GMT):
Hi,
I am getting below error while installing chaincode from the peer.
Command - kubectl exec $PEER_POD -n peers -- bash -c 'peer chaincode install -n mycc -v 1.0 -p /var/chaincode/'
Error - error getting chaincode code mycc: command : failed with error: "exec: "go": executable file not found in $PATH"
NOTE - I have placed the chaincode in the docker file system
Do I need to install GO in the peer container or machine, please suggest.
followed @nicolapaoli and @alexvicegrab 's session .
raj_shekhar (Fri, 29 Mar 2019 04:40:00 GMT):
Hi,
I am getting below error while installing chaincode from the peer.
Command - kubectl exec $PEER_POD -n peers -- bash -c 'peer chaincode install -n mycc -v 1.0 -p /var/chaincode/'
Error - error getting chaincode code mycc: command : failed with error: "exec: "go": executable file not found in $PATH"
NOTE - I have placed the chaincode in the docker file system
Do I need to install GO in the peer container or machine, please suggest.
Thanks,
yacovm (Sun, 31 Mar 2019 19:18:02 GMT):
https://chat.hyperledger.org/channel/fabric-kubernetes?msg=NfPPxQDZWW3zE2tXs
yacovm (Sun, 31 Mar 2019 19:18:28 GMT):
@iramiller it's here! (it is in v1.4.1 release candidate 1) you can check it out
klkumar369 (Tue, 02 Apr 2019 18:40:36 GMT):
Has joined the channel.
iramiller (Tue, 02 Apr 2019 21:15:31 GMT):
@yacovm :thumbsup:
raj_shekhar (Wed, 03 Apr 2019 08:43:02 GMT):
Hi
I have deployed the fabric-cli in K8s and able to install the go chaincode on peer.
But while instantiating the chaincode I am getting below error-
Error: could not assemble transaction, err proposal response was not successful, error code 500,
msg error starting container: error starting container: Failed to generate platform-specific docker build:
Error returned from build: 1 "can't load package: package ./github.com/chaincode: cannot find package "./github.com/chaincode" in:
/github.com/chaincode
raj_shekhar (Wed, 03 Apr 2019 08:43:02 GMT):
Hi @Privet-mir,
I have deployed the fabric-cli in K8s and able to install the go chaincode on peer.
But while instantiating the chaincode I am getting below error-
Error: could not assemble transaction, err proposal response was not successful, error code 500,
msg error starting container: error starting container: Failed to generate platform-specific docker build:
Error returned from build: 1 "can't load package: package ./github.com/chaincode: cannot find package "./github.com/chaincode" in:
/github.com/chaincode
raj_shekhar (Wed, 03 Apr 2019 10:53:19 GMT):
@iramiller
sahilgoel (Wed, 03 Apr 2019 10:58:55 GMT):
Has joined the channel.
raj_shekhar (Wed, 03 Apr 2019 11:10:32 GMT):
it is resolved , issue was related to relative paths
vanitas92 (Fri, 05 Apr 2019 14:19:39 GMT):
Hello everyone! I am trying to set up a multiple orderer with raft consensus with kubernetes. I have managed to bootstrap them and it seems that the first chain `testchainid` seems to connect with each orderer node but after somewhile it says that its no longer a TLS handshake happening. Some logs here when starting the orderer nodes:
```
2019-04-05 13:59:02.200 UTC [orderer.consensus.etcdraft] Start -> INFO 32d Starting Raft node channel=testchainid node=1
2019-04-05 13:59:02.200 UTC [orderer.common.cluster] Configure -> INFO 32e Entering, channel: testchainid, nodes: [ID: 2,
Endpoint: orderer1-allianz-technology-com:7050,
ServerTLSCert:-----BEGIN CERTIFICATE-----
MIICiTCCAjCgAwIBAgIRAL0S5....
ID: 3,
Endpoint: orderer2-allianz-technology-com:7050,
ServerTLSCert:-----BEGIN CERTIFICATE-----
...
2019-04-05 13:59:02.200 UTC [orderer.common.cluster] updateStubInMapping -> INFO 32f Allocating a new stub for node 2 with endpoint of orderer1-allianz-technology-com:7050 for channel testchainid
2019-04-05 13:59:02.200 UTC [orderer.common.cluster] updateStubInMapping -> INFO 330 Deactivating node 2 in channel testchainid with endpoint of orderer1-allianz-technology-com:7050 due to TLS certificate change
...
2019-04-05 13:59:02.200 UTC [orderer.common.cluster] func1 -> DEBU 331 Connecting to ID: 2,
Endpoint: orderer1-allianz-technology-com:7050,
...
for channel testchainid
2019-04-05 13:59:02.201 UTC [grpc] DialContext -> DEBU 332 parsed scheme: ""
2019-04-05 13:59:02.201 UTC [grpc] DialContext -> DEBU 333 scheme "" not registered, fallback to default scheme
2019-04-05 13:59:02.201 UTC [orderer.common.cluster] updateStubInMapping -> INFO 334 Allocating a new stub for node 3 with endpoint of orderer2-allianz-technology-com:7050 for channel testchainid
2019-04-05 13:59:02.201 UTC [orderer.common.cluster] updateStubInMapping -> INFO 335 Deactivating node 3 in channel testchainid with endpoint of orderer2-allianz-technology-com:7050 due to TLS certificate change
2019-04-05 13:59:02.201 UTC [orderer.common.cluster] func1 -> DEBU 336 Connecting to ID: 3,
Endpoint: orderer2-allianz-technology-com:7050,
...
for channel testchainid
2019-04-05 13:59:02.201 UTC [grpc] DialContext -> DEBU 337 parsed scheme: ""
2019-04-05 13:59:02.201 UTC [grpc] DialContext -> DEBU 338 scheme "" not registered, fallback to default scheme
2019-04-05 13:59:02.201 UTC [orderer.common.cluster] applyMembershipConfig -> INFO 339 2 exists in both old and new membership for channel testchainid , skipping its deactivation
2019-04-05 13:59:02.201 UTC [orderer.common.cluster] applyMembershipConfig -> INFO 33a 3 exists in both old and new membership for channel testchainid , skipping its deactivation
2019-04-05 13:59:02.201 UTC [orderer.common.cluster] Configure -> INFO 33b Exiting
2019-04-05 13:59:02.201 UTC [orderer.consensus.etcdraft] start -> DEBU 33c Starting raft node: #peers: 3 channel=testchainid node=1
2019-04-05 13:59:02.201 UTC [orderer.consensus.etcdraft] start -> INFO 33d Restarting raft node channel=testchainid node=1
2019-04-05 13:59:02.201 UTC [orderer.consensus.etcdraft] becomeFollower -> INFO 33e 1 became follower at term 1 channel=testchainid node=1
2019-04-05 13:59:02.201 UTC [orderer.consensus.etcdraft] newRaft -> INFO 33f newRaft 1 [peers: [], term: 1, commit: 3, applied: 0, lastindex: 3, lastterm: 1] channel=testchainid node=1
2019-04-05 13:59:02.201 UTC [orderer.common.server] Start -> INFO 340 Starting orderer:
Version: 1.4.1-rc1
Commit SHA: 29433f0
Go version: go1.11.5
OS/Arch: linux/amd64
2019-04-05 13:59:02.201 UTC [orderer.common.server] Start -> INFO 341 Beginning to serve requests
```
vanitas92 (Fri, 05 Apr 2019 14:21:53 GMT):
Until here it seems fine, after that, this happens:
```
2019-04-05 13:59:02.202 UTC [grpc] watcher -> DEBU 342 ccResolverWrapper: sending new addresses to cc: [{orderer1-allianz-technology-com:7050 0
akshay.sood (Sat, 06 Apr 2019 06:56:41 GMT):
Hi Fabric & k8s experts
I have a connection profile
`grpc://
vanitas92 (Mon, 08 Apr 2019 10:17:38 GMT):
Ok guys fixed it was a typo in my docker environment variables thx
iramiller (Mon, 08 Apr 2019 21:06:01 GMT):
@akshay.sood do you have the service name in your TLS certificates? We are using Kubernetes services for all of our endpoints
akshay.sood (Tue, 09 Apr 2019 02:17:42 GMT):
@iramiller tls is disabled
deelthor (Tue, 09 Apr 2019 12:06:34 GMT):
Has joined the channel.
deelthor (Tue, 09 Apr 2019 12:15:33 GMT):
Hey guys! We are running Hyperledger Composer and Hyperledger Fabric on Kubernetes. When Upgrading the Chaincode we suddenly get the error: `✖ Upgrading business network definition. This may take a minute...
Error: Error trying to upgrade business network. Error: No valid responses from any peers.
Response from attempted peer comms was an error: Error: failed to execute transaction bb7a720bccc7b646c4fa57774d0f72ae9e52898f364321b78a87c811ae4ffe30: error starting container: error starting container: cannot connect to Docker endpoint
Response from attempted peer comms was an error: Error: failed to execute transaction bb7a720bccc7b646c4fa57774d0f72ae9e52898f364321b78a87c811ae4ffe30: error starting container: error starting container: cannot connect to Docker endpoint` . Any suggestions are very welcome.
deelthor (Tue, 09 Apr 2019 12:15:33 GMT):
Hey guys! We are running Hyperledger Composer and Hyperledger Fabric on Kubernetes. When Upgrading the Chaincode we suddenly get the error: `✖ Upgrading business network definition. This may take a minute...
Error: Error trying to upgrade business network. Error: No valid responses from any peers.
Response from attempted peer comms was an error: Error: failed to execute transaction bb7a720bccc7b646c4fa57774d0f72ae9e52898f364321b78a87c811ae4ffe30: error starting container: error starting container: cannot connect to Docker endpoint
Response from attempted peer comms was an error: Error: failed to execute transaction bb7a720bccc7b646c4fa57774d0f72ae9e52898f364321b78a87c811ae4ffe30: error starting container: error starting container: cannot connect to Docker endpoint` . Any suggestions are very welcome.
deelthor (Tue, 09 Apr 2019 12:15:33 GMT):
Hey guys! We are running Hyperledger Composer and Hyperledger Fabric on Kubernetes. When Upgrading the Chaincode we suddenly get the error: `Upgrading business network definition. This may take a minute...
Error: Error trying to upgrade business network. Error: No valid responses from any peers.
Response from attempted peer comms was an error: Error: failed to execute transaction bb7a720bccc7b646c4fa57774d0f72ae9e52898f364321b78a87c811ae4ffe30: error starting container: error starting container: cannot connect to Docker endpoint
Response from attempted peer comms was an error: Error: failed to execute transaction bb7a720bccc7b646c4fa57774d0f72ae9e52898f364321b78a87c811ae4ffe30: error starting container: error starting container: cannot connect to Docker endpoint` . Any suggestions are very welcome.
deelthor (Tue, 09 Apr 2019 12:15:33 GMT):
Hey guys! We are running Hyperledger Composer and Hyperledger Fabric on Kubernetes. When Upgrading the Chaincode we suddenly get the error: ```Upgrading business network definition. This may take a minute...
Error: Error trying to upgrade business network. Error: No valid responses from any peers.
Response from attempted peer comms was an error: Error: failed to execute transaction bb7a720bccc7b646c4fa57774d0f72ae9e52898f364321b78a87c811ae4ffe30: error starting container: error starting container: cannot connect to Docker endpoint
Response from attempted peer comms was an error: Error: failed to execute transaction bb7a720bccc7b646c4fa57774d0f72ae9e52898f364321b78a87c811ae4ffe30: error starting container: error starting container: cannot connect to Docker endpoint``` . Any suggestions are very welcome.
LeeCherry (Tue, 09 Apr 2019 22:05:09 GMT):
Has joined the channel.
iramiller (Tue, 09 Apr 2019 22:42:56 GMT):
@akshay.sood what type of service are you using? Does your kubernetes service definition include a load balancer? The following is from a working service in my environment
```
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2018-07-26T00:10:59Z"
labels:
service: peer-COMPANYNAME
name: peer-0
namespace: COMPANYNAME
resourceVersion: "00000000"
selfLink: /api/v1/namespaces/COMPANY/services/peer-0
uid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
spec:
clusterIP: x.x.x.x
externalTrafficPolicy: Cluster
ports:
- name: api
nodePort: 32051
port: 7051
protocol: TCP
targetPort: 7051
```
akshay.sood (Wed, 10 Apr 2019 08:40:36 GMT):
@iramiller I tried with `clusterIP`
akshay.sood (Wed, 10 Apr 2019 08:40:40 GMT):
but it didnt worked
akshay.sood (Wed, 10 Apr 2019 08:40:50 GMT):
can you send me your connection.js ?
AndresMartinezMelgar.itcl (Wed, 10 Apr 2019 09:26:43 GMT):
@akshay.sood what do you pretend? I am using ClusterIp, when pods only comuncated inside cluster, and load balancer when they comunicated outside
AndresMartinezMelgar.itcl (Wed, 10 Apr 2019 09:26:43 GMT):
@akshay.sood what do you want to do? I am using ClusterIp, when pods only comuncated inside cluster, and load balancer when they comunicated outside
AndresMartinezMelgar.itcl (Wed, 10 Apr 2019 09:26:43 GMT):
@akshay.sood what do you want to do? I am using ClusterIp, when pods only comuncated inside cluster, and load balancer when they comunicate outside
iramiller (Wed, 10 Apr 2019 14:49:36 GMT):
@akshay.sood I recommend starting with a "cli" pod in kubernetes and using the `peer` and `discover` command line programs to test out your network before troubleshooting further. We use a pod with various configuration and certificates/msps stored in a PVC for management tasks (such as joining new members, upgrading/installing chaincode. This setup has provide very helpful for a divide and conquer approach to troubleshooting internal vs external connectivity issues. We use slightly customized version of hyperledger-fabric-tools:amd64-1.4.0 for this purpose.
AndresMartinezMelgar.itcl (Wed, 10 Apr 2019 16:03:49 GMT):
Hi @iramiller how work that "peer discover"
iramiller (Wed, 10 Apr 2019 16:05:02 GMT):
`discover --configFile /tmp/savedconfig.yaml config --channel CHANNELNAME --server peer-0.NAMESPACE_NAME:7051`
iramiller (Wed, 10 Apr 2019 16:05:44 GMT):
https://hyperledger-fabric.readthedocs.io/en/master/discovery-cli.html
iramiller (Wed, 10 Apr 2019 16:06:59 GMT):
From a "cli" pod using `kubectl exec -it cli-pod-name-here -- bash` you can run the above command (subject to your config, etc -- if you need help with that I recommend reading the docs above)
iramiller (Wed, 10 Apr 2019 16:10:56 GMT):
`discover --configFile /tmp/savedconfig.yaml --peerTLSCA msp/tlscacerts/ca.pem --userKey msp/keystore/admin\@ORGNAME-key.pem --userCert msp/admincerts/admin\@ORGNAME-cert.pem --MSP root.ORGNAME saveConfig` will create the saved config if you have the right certs/setup ... look for the details in that docs page ...
AndresMartinezMelgar.itcl (Wed, 10 Apr 2019 16:26:36 GMT):
ohh, thx tomorrow i'll prove it
mwagner (Wed, 10 Apr 2019 17:59:15 GMT):
Does the Fabric 2.0 Alpha include support fr k8s or is it stil the same bringup ?
NicolasHuray (Wed, 10 Apr 2019 19:47:05 GMT):
@mwagner Fabric is designed to run on any environment so I don't think the community is specially targeting Kubernetes. Nevertheless, the proposal for deploying chaincodes in cloud native environment is a step in the right direction to streamline chaincodes deployment and should bring scalability and performance as well. Please read the document for more info : https://docs.google.com/document/d/14l-0jjxw0SLrkpgEuXr0ZxAn0BtTJAQTupXvVkeIL2s/edit
mondraymond (Wed, 10 Apr 2019 20:39:16 GMT):
Has joined the channel.
raj_shekhar (Thu, 11 Apr 2019 08:58:47 GMT):
Hi @akshay.sood
I am also trying to connect to the fabric on Kubernetes cluster from Fabric-SDK, I am done with registering with CA and now I am trying to query the already installed sample chaincode but there I am getting rerror -- "error: [Remote.js]: Error: Failed to connect before the deadline URL:grpc://IP:7051
Error: Failed to connect before the deadline URL:grpc://IP:7051"
I am xposing my peer as a service in K8s can u plz suggest how to send grpc calls to peer in K8s??
raj_shekhar (Thu, 11 Apr 2019 09:00:48 GMT):
feel free to chime in if anybody has gone through same ,,
raj_shekhar (Thu, 11 Apr 2019 12:00:41 GMT):
resolved it,,,, firewall rules were not getting reflected
LeeCherry (Thu, 11 Apr 2019 13:03:59 GMT):
following Production set up as shown on this : https://github.com/aidtechnology/hgf-k8s-workshop
Pod won't start after installing the hlf-ca chart - anyone had this problem? Error is
Warning FailedScheduling 11m (x4 over 12m) default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 3 times)
iramiller (Thu, 11 Apr 2019 14:36:06 GMT):
@LeeCherry you need to look at your list of PVCs in your cluster... you have something wrong with your storage provider/capacity which is causing a PVC to not bind to a PV.
LeeCherry (Thu, 11 Apr 2019 14:53:03 GMT):
Well, i'm new to k8s, but looking at the dashboard, it all looks ok, 1gib in size as the chart provisioned and a big green tick :-)
LeeCherry (Thu, 11 Apr 2019 15:18:22 GMT):
The PVC issue has gone away after deleting the helm chart, the other issue is that the dependant postgres db wants 256 memory and my memory limit "kubectl describe limitrange memory-limit-range" is 128. Does postgres really need 256?
LeeCherry (Thu, 11 Apr 2019 15:29:49 GMT):
For anyone else with this issue, I fixed it with https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/
akshay.sood (Mon, 15 Apr 2019 04:20:42 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=crJDh57LATvdThB63) @raj_shekhar same here @raj_shekhar. Fabric-ca works fine because it is using http protocol. but peers, orderers are using grpc. I am having same issue but could not resolve or research further because of time limitation. If anyone know the solution please let us know
raj_shekhar (Mon, 15 Apr 2019 06:03:42 GMT):
@akshay.sood the grpc calls to query and instantiate the chaincode are working fine with HTTP protocol (for now I have used load balancer to expose it). ..
raj_shekhar (Mon, 15 Apr 2019 06:10:07 GMT):
but now I am getting the Prinicipal deserializtion failure.....
raj_shekhar (Mon, 15 Apr 2019 06:10:07 GMT):
but now I am getting the Prinicipal deserializtion failure..... some cert issue,,,,
akshay.sood (Mon, 15 Apr 2019 06:21:30 GMT):
can you share your connection profile?
AndresMartinezMelgar.itcl (Mon, 15 Apr 2019 07:16:02 GMT):
Hello, I am trying to create a network in kubernetes with kafka and 2 nodes to order.
When I try to create a channel it gives me this error: Error: got unexpected status: BAD_REQUEST - error authorizing update: error validating DeltaSet: policy for [Group] / Channel / Application not satisfied: Failed to reach implicit threshold of 1 sub-policies, required 1 remainin.
Does anyone know the reason for this error?
PS: restarting the network does not work, I've already tried it. The channel does not exist, I'm starting the network from 0
raj_shekhar (Mon, 15 Apr 2019 08:46:42 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=u9hHknt9hkWpmX9mD) @AndresMartinezMelgar.itcl set the peer admin identity and then try creating the channel
AndresMartinezMelgar.itcl (Mon, 15 Apr 2019 09:06:01 GMT):
Thanks!
raj_shekhar (Mon, 15 Apr 2019 09:08:41 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=mjQdWPQZgwjf5QJyf) @akshay.sood ok... will share
duwenhui (Tue, 16 Apr 2019 04:00:52 GMT):
QQ图片20190416115926.png
duwenhui (Tue, 16 Apr 2019 04:02:04 GMT):
Anyone who can help me solove this issue? How to expose those ports out of the Cluster?
AndresMartinezMelgar.itcl (Tue, 16 Apr 2019 06:05:28 GMT):
Clipboard - April 16, 2019 8:05 AM
AndresMartinezMelgar.itcl (Tue, 16 Apr 2019 06:05:28 GMT):
Clipboard - April 16, 2019 8:05 AM
AndresMartinezMelgar.itcl (Tue, 16 Apr 2019 06:10:47 GMT):
Clipboard - April 16, 2019 8:10 AM
AndresMartinezMelgar.itcl (Tue, 16 Apr 2019 06:10:59 GMT):
@duwenhui check that examples
duwenhui (Tue, 16 Apr 2019 06:14:30 GMT):
How do I set ClusterIP? @AndresMartinezMelgar.itcl
AndresMartinezMelgar.itcl (Tue, 16 Apr 2019 06:18:55 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=EKLvuuY7Qdr63RcqL) look this
duwenhui (Tue, 16 Apr 2019 06:23:58 GMT):
I means expose pots outside cluster. How to set spec.clusterIP 10.11.242.213? What IP can I set?
AndresMartinezMelgar.itcl (Tue, 16 Apr 2019 06:28:38 GMT):
@duwenhui
then look at the 2nd photo. There is a value called clusterip, which refers to the ip inside the cluster. It also lets you point to a specific port.
When you put loadBalancer and execute the service it returns a public IP from which you can access from outside the cluster
esaygi (Tue, 16 Apr 2019 09:15:08 GMT):
Has joined the channel.
aatkddny (Tue, 16 Apr 2019 12:17:39 GMT):
If you *just* want a nodeport, something simple like this will work
```
apiVersion: v1
kind: Service
metadata:
name: peer0-myorg
spec:
type: NodePort
ports:
- name: listen-endpoint
protocol: TCP
nodePort: 30011
port: 7051
targetPort: 7051
- name: profile-endpoint
protocol: TCP
nodePort: 30012
port: 9443
targetPort: 9443
selector:
app: peer0-myorg
```
Exposes 7051 on 30011 and the metrics endpoint on 9443 on 30012. You can hit any of the ip addresses of the nodes in your cluster and it should just work.
aatkddny (Tue, 16 Apr 2019 12:17:39 GMT):
If you *just* want a nodeport, something simple like this will work
```
apiVersion: v1
kind: Service
metadata:
name: peer0-myorg
spec:
type: NodePort
ports:
- name: listen-endpoint
protocol: TCP
nodePort: 30011
port: 7051
targetPort: 7051
- name: profile-endpoint
protocol: TCP
nodePort: 30012
port: 9443
targetPort: 9443
selector:
app: peer0-myorg
```
Exposes 7051 on 30011 and the metrics endpoint on 9443 on 30012. You can hit any of the ip addresses of the nodes in your cluster and it should just work.
Assumes your selector on the peer is `app: peer0-myorg` obviously. Change as required
iramiller (Tue, 16 Apr 2019 15:22:41 GMT):
@duwenhui for external traffic routing you need to be looking at a loadbalancer and not nodeport... technically a load balancer will likely use node ports and iptables underneath and you are certainly welcome to implement your own load balancer from scratch but I suspect this is not what you are looking for. Additionally I would not recommend exposing that operations endpoint (9443) to the public internet. If you are just attempting to connect to this peer for development purposes from outside of your cluster have a look at the `kubefwd` project.
duwenhui (Wed, 17 Apr 2019 07:10:31 GMT):
@iramiller @aatkddny @AndresMartinezMelgar.itcl Thanks for Responding to my question. I hava solved this problem . In fact, the problem I met is about deployment. If you set spec.ClusterIP: None at the start with helm, Whaterver you modify the config online, k8s don't allow this. So I delete service, delete spec.ClusterIP, set type: NodePort and It works fine.
duwenhui (Wed, 17 Apr 2019 07:13:32 GMT):
I will try to use kubefwd.
ygnr (Wed, 17 Apr 2019 23:38:39 GMT):
How much memory is each peer consuming for you guys on Kubernetes?
raj_shekhar (Fri, 19 Apr 2019 07:18:00 GMT):
Hi,
anybody got this error while invoking chaincode from sdk --
peer logs - "Principal deserialization failure (the supplied identity is not valid: x509: certificate signed by unknown authority) for identity 0"
raj_shekhar (Fri, 19 Apr 2019 07:18:00 GMT):
Hi,
anybody got this error while invoking chaincode from sdk ??--
peer logs - "Principal deserialization failure (the supplied identity is not valid: x509: certificate signed by unknown authority) for identity 0"
raj_shekhar (Fri, 19 Apr 2019 09:19:26 GMT):
I am using fabric-ca to generate all the identities
iramiller (Mon, 22 Apr 2019 18:38:35 GMT):
@ygnr peer memory usage depends heavily on the client API connecting to it and that programs use of connections (the Java SDK has caused us some challenges at times). Additionally the usage patterns matter quite a bit ... adding new peer with a 100k blocks will be a much different load than 100s or 1000s of blocks.
iramiller (Mon, 22 Apr 2019 18:39:41 GMT):
That said we have been able to function very well with a range of memory utilization between 2-4GB (includes couch container which is in the same pod as our peer container)... during intensive activities our memory usage will trend as high as 10-20GB.
raj_shekhar (Tue, 23 Apr 2019 10:13:09 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=uP4rzofx6SEQmDabs) @iramiller can u suggest something on this?
ShwetaTripathi (Wed, 24 Apr 2019 05:22:52 GMT):
Has joined the channel.
ShwetaTripathi (Wed, 24 Apr 2019 05:31:51 GMT):
Hi, Can someone help me with the procedure to deploy Hyperledger Fabric network on Kubernetes cluster?
raj_shekhar (Wed, 24 Apr 2019 05:45:28 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=uP4rzofx6SEQmDabs) @dave.enyeart can u please suggest on this
raj_shekhar (Wed, 24 Apr 2019 05:45:28 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=uP4rzofx6SEQmDabs) can u please suggest on this
raj_shekhar (Wed, 24 Apr 2019 10:38:24 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=a8rGGM79vshsZMedg) Resolved it,,,,,,, It was a issue of default policies and capabilities
AndresMartinezMelgar.itcl (Thu, 25 Apr 2019 07:30:19 GMT):
what is the different between HFClient and FabricClient classes?
Dhiraj1990 (Fri, 26 Apr 2019 04:08:21 GMT):
Has joined the channel.
Dhiraj1990 (Fri, 26 Apr 2019 04:11:51 GMT):
Hello all,
I have created orderer and cli pods. When i go to cli shell and create channel then it is not able to connect with ordrer.
Error: failed to create deliver client: orderer client failed to connect to orderer:7050: failed to create new connection: context deadline exceeded
The port for order is open and when i go to orderer shell and do telnet localhost 7050 it is connected but when specify the ip for pod then it does not work
Dhiraj1990 (Fri, 26 Apr 2019 04:11:51 GMT):
Hello all,
I have created orderer and cli pods. When i go to cli shell and create channel then it is not able to connect with ordrer.
*Error: failed to create deliver client: orderer client failed to connect to orderer:7050: failed to create new connection: context deadline exceeded*
The port for order is open and when i go to orderer shell and do `telnet localhost 7050` it is connected but when specify the ip for pod then it does not work
Dhiraj1990 (Fri, 26 Apr 2019 04:11:51 GMT):
Hello all,
I have created orderer and cli pods. When i go to cli shell and create channel then it is not able to connect with ordrer.
*Error: failed to create deliver client: orderer client failed to connect to orderer:7050: failed to create new connection: context deadline exceeded*
The port for order is open and when i go to orderer shell and do `telnet localhost 7050` it is connected but when specify the ip for pod then it does not work.
I am using Google Cloud for deloyment
Dhiraj1990 (Fri, 26 Apr 2019 04:11:51 GMT):
Hello all,
I have created orderer and cli pods. When i go to cli shell and create channel then it is not able to connect with ordrer.
*Error: failed to create deliver client: orderer client failed to connect to orderer:7050: failed to create new connection: context deadline exceeded*
The port for order is open and when i go to orderer shell and do `telnet localhost 7050` it is connected but when specify the ip for pod then it does not work.
I am using Google Cloud for deloyment. I have also added firewall rules for ingress and egress for all IP and all Ports.
Dhiraj1990 (Fri, 26 Apr 2019 04:11:51 GMT):
Hello all,
I have created orderer and cli pods. When i go to cli shell and create channel then it is not able to connect with ordrer.
*Error: failed to create deliver client: orderer client failed to connect to orderer:7050: failed to create new connection: context deadline exceeded*
The port for order is open and when i go to orderer shell and do `telnet localhost 7050` it is connected but when specify the ip for pod then it does not work.
*I am using Google Cloud for deloyment. I have also added firewall rules for ingress and egress for all IP and all Ports.*
raj_shekhar (Fri, 26 Apr 2019 05:05:25 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=T6n47xB362MevC5iD) @Dhiraj1990 how you exposed ur orderer endpoints?
raj_shekhar (Fri, 26 Apr 2019 05:05:25 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=T6n47xB362MevC5iD) @Dhiraj1990 how you exposed ur orderer endpoints in K8s?
Dhiraj1990 (Fri, 26 Apr 2019 05:05:55 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=u8yiWinggmNmnjud2) @raj_shekhar I have exposed 7050 port as host port. Also through cli it is able to communicate because it does not show could not resolve host.
Dhiraj1990 (Fri, 26 Apr 2019 05:05:55 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=u8yiWinggmNmnjud2) @raj_shekhar I have exposed 7050 port as host port
raj_shekhar (Fri, 26 Apr 2019 05:11:35 GMT):
as this group is for fabric on kubernetes so I thought u r doing it on GKE
raj_shekhar (Fri, 26 Apr 2019 05:12:36 GMT):
ping the command u r using to create channel
Dhiraj1990 (Fri, 26 Apr 2019 05:14:29 GMT):
@raj_shekhar
Dhiraj1990 (Fri, 26 Apr 2019 05:14:30 GMT):
org1.example.com/peers/peer1.org1.example.com/tls
Dhiraj1990 (Fri, 26 Apr 2019 05:14:39 GMT):
peer channel create -o orderer:7050 -c $CHANNEL_NAME -f ./crypto-config/channel-artifacts/channel.tx --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/crypto-config/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem
raj_shekhar (Fri, 26 Apr 2019 05:36:02 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=MrajmzzjSdDSa4E5i) @Dhiraj1990 as per our discussion , it looks rancher service issue , the endpoints are not exposed properly,,,,
Anybody in group having rancher hands on feel free to chip in ....
Fias (Fri, 26 Apr 2019 06:54:46 GMT):
Has joined the channel.
Dhiraj1990 (Fri, 26 Apr 2019 07:00:58 GMT):
Hello all ,
On cli when i run
`peer channel list`
i get below error
`Error: Failed sending proposal, got rpc error: code = Unknown desc = access denied: channel [] creator org [Org1MSP]`
And when i am checking logs for peer0
`MSP error: the supplied identity is not valid: x509: certificate signed by unknown authority`
What is the issue here. seems like certificate is not valid but i have generated them as per docs. I have deployed network on kubernetes
iramiller (Fri, 26 Apr 2019 22:31:26 GMT):
@Dhiraj1990 make sure you are passing the correct CA cert arg.
iramiller (Fri, 26 Apr 2019 22:32:06 GMT):
```
Flags:
--cafile string Path to file containing PEM-encoded trusted certificate(s) for the ordering endpoint
```
iramiller (Fri, 26 Apr 2019 22:33:06 GMT):
and that any client auth certificates you are using are signed by a CA known to the peer.
iramiller (Fri, 26 Apr 2019 22:34:31 GMT):
And last ... you would have more luck with these types of questions that are not specific to the Kubernetes environment and the operation of Hyperledger Fabric in Kubernetes by posting them in a more appropriate channel.
AndresMartinezMelgar.itcl (Sat, 27 Apr 2019 19:09:34 GMT):
hello, in order to install and instantiate a java chaincode within the kubernetes cluster, do I need to have a "fabric-javaenv"?
awattez (Sun, 28 Apr 2019 20:25:45 GMT):
Has joined the channel.
mwagner (Mon, 29 Apr 2019 03:59:26 GMT):
permission denied
LeeCherry (Mon, 29 Apr 2019 10:38:51 GMT):
I'm following the AID:Tech Hyperledger on k8s but I can't get this command to work
FABRIC_CA_CLIENT_HOME=./config fabric-ca-client enroll -u https://ord-admin:OrdAdm1nPW@$CA_INGRESS -M ./OrdererMSP - I get the error "fabric-ca-client: command not found". Anyone able to point me in the right direction? I'm new to this
LeeCherry (Mon, 29 Apr 2019 10:40:17 GMT):
at 16.13 it looks so easy :-) https://www.youtube.com/watch?v=3tVk7yrGSSE
LeeCherry (Mon, 29 Apr 2019 10:40:17 GMT):
at 16.13 it looks so easy :-) https://www.youtube.com/watch?v=3tVk7yrGSSE @alexvicegrab
LeeCherry (Mon, 29 Apr 2019 10:41:15 GMT):
leecherry
MohammedR (Mon, 29 Apr 2019 10:53:47 GMT):
@LeeCherry you need fabric-ca-client binary to enroll user
LeeCherry (Mon, 29 Apr 2019 10:55:19 GMT):
Thanks @MohammedR , where can I get that from?
MohammedR (Mon, 29 Apr 2019 10:57:33 GMT):
Refer this link to download binaries https://hyperledger-fabric.readthedocs.io/en/release-1.4/install.html
LeeCherry (Mon, 29 Apr 2019 12:21:08 GMT):
Great thanks, got the command working
LeeCherry (Mon, 29 Apr 2019 12:21:08 GMT):
Great thanks @MohammedR , got the command working
LeeCherry (Mon, 29 Apr 2019 12:21:46 GMT):
However, get this error, x509: certificate is valid for ingress.local, not ca.dev.blockchain.cherry.com
Must have missed a step or something earlier
AshishMishra 1 (Tue, 30 Apr 2019 06:10:39 GMT):
Hi experts, I have setup a fabric cluster on kubernetes using helm charts. I could join the peers to a channel and install chaincode on it. However when I 'm instantiating the chaincode, I am not getting any error the chaincode container also starts successfully on the peer node but it's not registering himself with the peer container somehow. So when I do peer chaincode list --instantiated -C mychannel... I don't see anything. Am I missing some env here?
pankajcheema (Thu, 02 May 2019 09:27:39 GMT):
https://stackoverflow.com/questions/55949232/unable-to-communicate-with-orderer-from-peer-hyperledger-fabric
pankajcheema (Thu, 02 May 2019 09:28:17 GMT):
Please have a look if anyone can suggest me on the same it would be a lot helpful for me .
JorgeNavarro (Thu, 02 May 2019 10:51:09 GMT):
Hello, im trying to deploy the web ui dashboard but when i use the following command: "kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/recommended/kubernetes-dashboard.yaml", it generates the following error: "kubernetes-dashboard-minimal is forbidden: attemp to grant extra privileges".
Any suggestions?
iramiller (Thu, 02 May 2019 15:54:41 GMT):
@JorgeNavarro you need to ensure that your current user that is applying that configuration has the appropriate RBAC roles granted. Your user must have all of the permissions listed that you are attempting to grant to that new service account.
iramiller (Thu, 02 May 2019 15:56:58 GMT):
The easiest way to get all of those rights is to make sure your user has ClusterAdmin (or equivalent role depending on your Kubernetes install). If you are not the owning system administrator (or group) then you are going to need to ask one of those members to create this for you.
Henrycoffin (Fri, 03 May 2019 07:18:23 GMT):
Has joined the channel.
Ramrockez143 (Mon, 13 May 2019 06:36:16 GMT):
Has joined the channel.
Ramrockez143 (Mon, 13 May 2019 06:36:17 GMT):
iam facing problem like below
Ramrockez143 (Mon, 13 May 2019 06:36:33 GMT):
2019-05-13 06:18:13.513 UTC [grpc] HandleSubConnStateChange -> DEBU 04c pickfirstBalancer: HandleSubConnStateChange: 0xc000239b60, READY
Error: got unexpected status: SERVICE_UNAVAILABLE -- backing Kafka cluster has not completed booting; try again later
root@cli-798764bc8b-bs7s7:/opt/gopath/src/github.com/hyperledger/fabric/peer# exit
Ramrockez143 (Mon, 13 May 2019 06:36:44 GMT):
while creating channel
Ramrockez143 (Mon, 13 May 2019 06:37:33 GMT):
here thing is that kafka is in different eks cluster and orderer in different k8s cluster
Ramrockez143 (Mon, 13 May 2019 06:44:36 GMT):
2019-05-12 19:14:30.328 UTC [orderer.consensus.kafka] setupProducerForChannel -> INFO 008 [channel: testchainid] Setting up the producer for this channel...
2019-05-13 04:20:00.724 UTC [orderer.common.broadcast] ProcessMessage -> WARN 009 [channel: telcodlt] Rejecting broadcast of message from 10.10.74.110:59948 with SERVICE_UNAVAILABLE: rejected by Consenter: backing Kafka cluster has not completed booting; try again later
Ramrockez143 (Mon, 13 May 2019 06:44:53 GMT):
above logs of orderer
MohammedR (Mon, 13 May 2019 08:57:14 GMT):
@Ramrockez143 orderer in not able to connect to kafka nodes check your endpoints, check kafka endpoints in configtx.yaml file
Ramrockez143 (Mon, 13 May 2019 09:12:49 GMT):
Hi Mohammed,
Ramrockez143 (Mon, 13 May 2019 09:13:00 GMT):
thank you.
Ramrockez143 (Mon, 13 May 2019 09:13:33 GMT):
endpoints are also proper
Ramrockez143 (Mon, 13 May 2019 09:14:34 GMT):
but kafka is in one eks cluster and remaining fabric is in different eks cluster
Ramrockez143 (Mon, 13 May 2019 09:17:28 GMT):
here is thing is that if kafka & orderer are in same cluster iam able to create channel creation and channel joining
Ramrockez143 (Mon, 13 May 2019 09:18:59 GMT):
you need to put cordns cluster ip in docker configuration default path
Ramrockez143 (Mon, 13 May 2019 09:21:23 GMT):
# Docker configuration
Since instantiating chaincode requires a container created by docker And this container needs to communicate with the peer svc in k8s, so additional configuration is required.
# Configure docker configuration to modify the docker option DOCKER_OPTS
Add --dns in the DOCKER_OPTS to add the ip of the dns in k8s
# Use DOCKER_OPTS to modify the daemon startup options in file /etc/default/docker
DOCKER_OPTS="--dns=10.96.0.10 --dns=172.17.0.1 --dns-search kubernetes --dns-search kubernetes.default --dns-search kubernetes.default.svc --dns-search kubernetes.default.svc.cluster.local --dns-search ec2.internal --dns-opt ndots:2 --dns-opt timeout:2 --dns-opt attempts:2"
MatasV (Mon, 13 May 2019 11:23:37 GMT):
Has joined the channel.
AshishMishra 1 (Tue, 14 May 2019 09:11:37 GMT):
@Ramrockez143 thanks. That worked, but having with mTLS with the same n/w. Have you ever tried mTLS with kubernetes?
AshishMishra 1 (Tue, 14 May 2019 09:45:52 GMT):
@iramiller and @yacovm very informative discussion thanks. @iramiller I wanted to have a similar setup, I did try with nginx ingress controller with cert manager to automatically manage the tls certs. Though I want a service mess but I feel it would be overkill for our small deployment. Nginx thing didn't work both while trying to terminate the tls at the ingress and even on the peer container. Always failed with a tls handshake error.
So I am thinking of giving isitio a try with tls being managed by istio/envoy. Does that work in your setup w/o code changes in peers/orderers? I am afraid for mTLS since the Kubernetes NATing might do something with the hostnames/Ips.
iramiller (Tue, 14 May 2019 14:24:03 GMT):
We are using Istio with the rest of our systems but have not integrated it with Hyperledger (yet). If I were going to attempt this I would use the Citadel to generate certificates and store them in a secret/configmap and mount this into the peer container. The peer would then start and use these certificates per the normal process.
The approach of using the sidecar will not work with the peer as the peer expects the secure communication to be terminated directly with the peer. The Istio mesh will work with the other end of the connection (depending on the caller) to inject mutual TLS certificates which the peer is expecting.
iramiller (Tue, 14 May 2019 14:25:01 GMT):
We are using Istio with the rest of our systems but have not integrated it with Hyperledger (yet). If I were going to attempt this I would use the Citadel to generate certificates and store them in a secret/configmap and mount this into the peer container. The peer would then start and use these certificates per the normal process.
The approach of using the sidecar will not work with the peer as the peer expects the secure communication to be terminated directly with the peer. The Istio mesh will work with the other end of the connection (depending on the caller) to inject mutual TLS certificates which the peer is expecting.
iramiller (Tue, 14 May 2019 14:26:15 GMT):
My current project is a custom BCCSP integration for HashiCorp Vault that provides certificate storage as well as PKI integration.
iramiller (Tue, 14 May 2019 14:29:10 GMT):
For our needs the Hyperledger Fabric project unfortunately does not align well with an enterprise cloud systems approach and we are continually applying quite a bit of software development effort to make it do these things (native Kubernetes chain code pods and infrastructure, real load balancing and failover, routing, advanced monitoring, etc).
iramiller (Tue, 14 May 2019 14:30:52 GMT):
For anyone else who wants to integrate Fabric into their environment in this way (strong Kubernetes support, Istio, Vault/PKI) I caution them to understand that it will be a significant amount of work that will be ongoing.
mauricio (Tue, 14 May 2019 14:32:36 GMT):
I agree @iramiller but also for a complex production solution it's completely necessary
iramiller (Tue, 14 May 2019 14:34:14 GMT):
It is my opinion that the majority of resources for the Hyperledger project are (or at least were) focused on small demonstration system projects that run on a developer workstation in docker and are not meant for long term production use.
mauricio (Tue, 14 May 2019 14:34:43 GMT):
I agree
iramiller (Tue, 14 May 2019 14:34:46 GMT):
There are simply far fewer users that need a large system than need to simply become familiar with blockchain for the first time
mauricio (Tue, 14 May 2019 14:35:13 GMT):
There isn't information about how to build a large production system with Fabric
mauricio (Tue, 14 May 2019 14:35:42 GMT):
It's hard, but also is our task see how can we help to improve this
iramiller (Tue, 14 May 2019 14:36:33 GMT):
Welcome to the bleeding edge ... paving the way requires a lot of work... those who come later will have a much easier time
mauricio (Tue, 14 May 2019 14:38:48 GMT):
hahaha yes, but after we'll can to help that people, and that is great. Also the people behind Hyperledger Fabric is making a great work.
AshishMishra 1 (Wed, 15 May 2019 04:46:18 GMT):
I think Fabric can very well cater to large production systems already. It has all the building blocks, its just that scaling the components being a permissioned blockchain is a pain and GRPC loadbalancing didnt work for me as I was expecting it to for the orderers, so everytime I add an orderer I need not update the channel defination.
raj_shekhar (Wed, 15 May 2019 08:14:32 GMT):
HI,
I have configured peer node to expose metrics on 127.0.0.1:9443 and I am able to access it from inside the peer container (wget 127.0.0.1/metrics).
I tried exposing it using loadbalancer service(by confguring 9443 port in service) but not able to access metrics.
HL Network is on Kubernetes.
Is there any better way to expose them to outside.
raj_shekhar (Wed, 15 May 2019 08:14:53 GMT):
I have configured the container port too in the peer deployment configruation
Ramrockez143 (Wed, 15 May 2019 09:02:55 GMT):
yes
AshishMishra 1 (Wed, 15 May 2019 10:31:27 GMT):
@Ramrockez143 Is it working? How are you authenticating the client against the server. Say my client here is the peer and orderer is the server. Also are you using any ingress controller? I am always getting a TLS handshake error.
Ramrockez143 (Wed, 15 May 2019 12:01:15 GMT):
no
iramiller (Wed, 15 May 2019 15:04:10 GMT):
It is absolutely possible--I am part of a several hundred million dollar blockchain company using Hyperledger as part of its solution. As you say, the building blocks are present and yet a deeply experienced team will be required to make them work. Further, in a few specific cases rewriting and replacing components of the system is required. And don't forget that a significant amount of operational support infrastructure will be needed for ongoing management (scripts, installation/configuration files, etc).
AshishMishra 1 (Thu, 16 May 2019 10:53:56 GMT):
Did anyone run Fabric 2.0 alpha on K8s yet? Etcd with tls pinning would be a challenge.
vanitas92 (Thu, 16 May 2019 11:13:37 GMT):
I have managed to deploy Fabric 1.4.1 with Raft consensus in K8s, which implements etcd only in TLS mode. Haven't got time to look for 2.0 alpha but does it make a huge difference with TLS pinning? I am not familiar with the concept of TLS pinning. Would that help?
yacovm (Thu, 16 May 2019 11:14:02 GMT):
@AshishMishra 1 why is TLS pinning a challenge?
AshishMishra 1 (Thu, 16 May 2019 11:29:58 GMT):
@vanitas92 etcd used TLS pinning by default. So you are already using it. :)
AshishMishra 1 (Thu, 16 May 2019 11:33:03 GMT):
@yacovm because I 've been already facing issues with mutual TLS in fabric. So was skeptical about etcd would go down easy with the tls.
yacovm (Thu, 16 May 2019 11:35:16 GMT):
what are the issues? do you have TLS termination?
AshishMishra 1 (Thu, 16 May 2019 12:03:39 GMT):
No, I 'm using a nginx ingress with nginx.ingress.kubernetes.io/backend-protocol: "GRPCS"
AshishMishra 1 (Thu, 16 May 2019 12:03:54 GMT):
which is supposed to proxy the grpc traffic directly to the peer container.
AshishMishra 1 (Thu, 16 May 2019 12:06:48 GMT):
and even my peer is having trouble to connect to itself... Like the peer channel list throws a tls error.
vanitas92 (Thu, 16 May 2019 12:19:06 GMT):
:sweat_smile: thx. Not really an expert in security hahaha
Ramrockez143 (Fri, 17 May 2019 05:31:03 GMT):
hi guys
Ramrockez143 (Fri, 17 May 2019 05:31:59 GMT):
i have one scenario like one orderer with 3 replica in hyperledger fabric in k8s .
Ramrockez143 (Fri, 17 May 2019 05:32:32 GMT):
can you suggest for that implementation
AndresMartinezMelgar.itcl (Fri, 17 May 2019 10:13:57 GMT):
@Ramrockez143 what do you want to do?
Ramrockez143 (Fri, 17 May 2019 12:07:16 GMT):
i wan to implement loadbalancer for orderer
iramiller (Fri, 17 May 2019 13:41:40 GMT):
@Ramrockez143 you can certainly stand up multiple orderers but you won't be making a kubernetes load balancer work in the traditional sense (one dns round robin across all available instances) due to issues with mTLS.
iramiller (Fri, 17 May 2019 13:42:58 GMT):
As many in the kubernetes channel have found the architecture choices in Hyperledger will make common cloud scaling technologies and techniques exceptionally difficult to implement.
iramiller (Fri, 17 May 2019 13:42:58 GMT):
As many in the kubernetes channel have found the architecture choices in Hyperledger Fabric will make common cloud scaling technologies and techniques exceptionally difficult to implement.
iramiller (Fri, 17 May 2019 13:44:48 GMT):
My recommendation would be to follow the design and setup guides and keep kubernetes aspects contained in the pod/container scheduling areas and try to stick with one-to-one mappings on the DNS side. (orderer-0.namespace, orderer-1.namespace, etc).
iramiller (Fri, 17 May 2019 13:45:25 GMT):
This approach will keep you in close alignment with the guides and software as released and make your life much easier.
circlespainter (Sat, 18 May 2019 07:40:42 GMT):
Has joined the channel.
Ramrockez143 (Tue, 21 May 2019 04:00:10 GMT):
hi guys
Ramrockez143 (Tue, 21 May 2019 04:02:29 GMT):
my scenario is orderer and peer are in different eks clusters,while instatiating chain code iam able create chaincode container but while invoking it showing that chaincode not found
Ramrockez143 (Tue, 21 May 2019 04:36:49 GMT):
while invoking iam getting following error
Ramrockez143 (Tue, 21 May 2019 04:36:53 GMT):
019-05-21 04:35:47.782 UTC [chaincodeCmd] chaincodeInvokeOrQuery -> DEBU 0ad ESCC invoke result: response:
MohammedR (Tue, 21 May 2019 04:47:10 GMT):
@Ramrockez143 chaincode container didn't build successfully check peer logs for more info
Ramrockez143 (Tue, 21 May 2019 05:02:04 GMT):
no errors in peer
Ramrockez143 (Tue, 21 May 2019 05:02:52 GMT):
orderers services are clusterIP
Ramrockez143 (Tue, 21 May 2019 05:03:05 GMT):
peer services are node port
vanitas92 (Tue, 21 May 2019 10:18:34 GMT):
you might need to put orderer services in nodeport since the peers need to contact the orderer service, as you said they are in different clusters
vanitas92 (Tue, 21 May 2019 10:18:34 GMT):
@Ramrockez143 you might need to put orderer services in nodeport since the peers need to contact the orderer service, as you said they are in different clusters
vanitas92 (Tue, 21 May 2019 10:18:34 GMT):
@Ramrockez143 during instantiate phase, the orderer needs to know what version of chaincode you are using i think
Ramrockez143 (Tue, 21 May 2019 11:26:56 GMT):
any body deployed hlf on multiple eks cluster
Ramrockez143 (Tue, 21 May 2019 11:27:23 GMT):
iam struggling with chaincode instantiation
iramiller (Tue, 21 May 2019 19:39:50 GMT):
@Ramrockez143 Yes. But I ended up making a fork of Hyperledger that schedules chain code pods directly instead of attempting to use a privileged docker call. This approach is pretty straight forward but isn't something that can be directly integrated into Fabric because it abandons the entire chaincode package/release/schedule approach in Fabric. Further our impl
iramiller (Tue, 21 May 2019 19:39:50 GMT):
@Ramrockez143 Yes. But I ended up making a fork of Hyperledger that schedules chain code pods directly instead of attempting to use a privileged docker call. This approach is pretty straight forward but isn't something that can be directly integrated into Fabric because it abandons the entire chaincode package/release/schedule approach in Fabric. Further our implementation is specific to our environment and our docker repositories ...
iramiller (Tue, 21 May 2019 19:40:45 GMT):
if you want to pursue that path I am happy to share pieces of the code and background...
iramiller (Tue, 21 May 2019 19:42:34 GMT):
in general though even though the approach is working very well for us it isn't an approach (managing a fabric fork) that I would recommend for most people. The 2.0 platform has improvements planned for this process which will hopefully make things better for everyone. These changes are not compatible with the approach we are currently using for Kubernetes and as such we will have to re-engineer our approach
JorgeNavarro (Wed, 22 May 2019 07:27:09 GMT):
Hello, if i want to use the function "GetHistoryForKey(key string)", i need to enable the environment variable "enableHistoryDatabase", where should i find this variable??
knagware9 (Wed, 22 May 2019 10:28:25 GMT):
this need to be enable in docker compose file or config.yaml file at peer container
Ramrockez143 (Wed, 22 May 2019 10:44:13 GMT):
Hi
Ramrockez143 (Wed, 22 May 2019 10:45:21 GMT):
how to add extra hosts to orderer and peer deployments in eks multiple clusters
Ramrockez143 (Wed, 22 May 2019 10:53:46 GMT):
how we can communicate two eks clusters
vanitas92 (Wed, 22 May 2019 11:12:46 GMT):
Hello channel. Has anyone tried to use Istio with Hyperledger fabric within a Kubernetes cluster. Also with different clouds communicating each other? If that is the case, how is your experience? Thank you very much!
lucky114407 (Wed, 22 May 2019 11:14:33 GMT):
Has joined the channel.
iramiller (Wed, 22 May 2019 14:30:18 GMT):
@vanitas92 we have 10 different production kubernetes clusters communicating across Azure, Amazon, and Google Kubernetes services. We use a private VPN solution implemented with Wireguard. We have explored Istio as well (and are using it with other services in our infrastructure) but we have not implemented it with Fabric at this time. Based on my initial research Istio would be severely limited in capability when integrated with Hyperledger as designed today. The TLS certificate management pieces of Istio would likely be the only real benefit at this point.
iramiller (Wed, 22 May 2019 14:30:18 GMT):
@vanitas92 we have 10 different production kubernetes clusters communicating across Azure, Amazon, and Google Kubernetes services. We use a private VPN solution implemented with Wireguard. We have explored Istio as well (and are using it with other services in our infrastructure) but we have not implemented it with Fabric at this time. Based on my initial research Istio would be severely limited in capability when integrated with Hyperledger as designed today.
iramiller (Wed, 22 May 2019 14:33:58 GMT):
While it is arguably more work we are not exposing any of the blockchain service endpoints (orderer, peer) directly to the internet but instead run custom API instances inside our network and expose those instead. This provides an interface we control, exposed over protocols that are friendly to the network load balancers, and run over Istio for enterprise level features.
iramiller (Wed, 22 May 2019 14:39:58 GMT):
The most useful integration capability that Istio could bring to Fabric (without extensive modifications) is probably certificate management. A separate evaluation of certificate management needs within our consortium have lead us to explore a Hashicorp Vault integrated with a custom MSP/BCCSP plugin.
iramiller (Wed, 22 May 2019 14:44:27 GMT):
My experience working directly within the Hyperledger code base on this plugin has lead me to believe that this type of work is a little bit ahead of the curve and will be substantially easier once the Fabric code base matures. The HLF team is well aware of these issues and is actively working to make things better: https://jira.hyperledger.org/browse/FAB-12246
levinem (Wed, 22 May 2019 16:43:10 GMT):
Has joined the channel.
yacovm (Wed, 22 May 2019 19:55:58 GMT):
@iramiller - in Fabric, the channel certificates used for verification of transactions are stored in the Blockchain, so you cannot use Istio or anything like that to manage them.
yacovm (Wed, 22 May 2019 19:56:31 GMT):
root / intermediate TLS CAs that are used to trust other organizations, are also (currently) derived from the Blockchain.
yacovm (Wed, 22 May 2019 19:57:34 GMT):
the whole point of Fabric is that it has decentralized management - you can make it so you can only modify the channel configuration if the majority of organizations vote on it
yacovm (Wed, 22 May 2019 19:57:50 GMT):
isn't Istio centrally managed?
yacovm (Wed, 22 May 2019 19:58:45 GMT):
another issue is that while Fabric uses x509 certificates _ now _, the MSP design doesn't tie you to use only x509 certificates
yacovm (Wed, 22 May 2019 20:00:52 GMT):
moreover - Istio is for microservices. Fabric is not a micro-service.....
yacovm (Wed, 22 May 2019 20:00:52 GMT):
moreover - Istio is for microservices. Fabric is not designed for micro services. I guess you can make the chaincode look like a micro-service if you really want
yacovm (Wed, 22 May 2019 20:01:59 GMT):
but the peers and orderers, are not micro-services.
iramiller (Wed, 22 May 2019 21:12:21 GMT):
@yacovm (1,2) the PUBLIC keys are stored in blockchain, how the PRIVATE keys are managed is the primary concern of those who would be interested in a Hashicorp Vault or Istio Citadel solution. (3) the member is free to manage their certificates in a secure and well controlled way while distributing the public keys for updates in the normal hyperledger way (votes on updates). (4) Istio is a useful method for securing and monitoring infrastructure communication. While I often describe the members in my network as 'mine' I use this characterization because I have defined the processing and architecture used by members. The members would be in control of their own instances of Istio.
iramiller (Wed, 22 May 2019 21:20:06 GMT):
@yacovm 'the peer/orderer processes are not microservices' ... they are also quite clearly not designed for high availability nor cloud environments either. Those of us that make the system work in these environments put quite a bit of effort into making these things happen. Based on https://jira.hyperledger.org/browse/FAB-13582 it would seem that chaincode containers are about to become servers (possibly implying a large proliferation of these instances if versioning of chaincode becomes fully supported). The more instances that are created the more an orchestration platform like kubernetes becomes a requiremewnt... and the more pieces that need to communication with the potential for issues that something like Istio helps diagnose.
iramiller (Wed, 22 May 2019 21:20:06 GMT):
@yacovm 'the peer/orderer processes are not microservices' ... they are also quite clearly not designed for high availability nor cloud environments either. Those of us that make the system work in these environments put quite a bit of effort into making these things happen. Based on https://jira.hyperledger.org/browse/FAB-13582 it would seem that chaincode containers are about to become servers (possibly implying a large proliferation of these instances if versioning of chaincode becomes fully supported). The more instances that are created the more an orchestration platform like kubernetes becomes a requirement... and the more pieces that need to communication with the potential for issues that something like Istio helps diagnose.
iramiller (Wed, 22 May 2019 21:20:06 GMT):
@yacovm 'the peer/orderer processes are not microservices' ... they are also quite clearly not designed for high availability nor cloud environments either. Those of us that make the system work in these environments put quite a bit of effort into making these things happen. Based on https://jira.hyperledger.org/browse/FAB-13582 it would seem that chaincode containers are about to become servers (possibly implying a large proliferation of these instances if versioning of chaincode becomes fully supported). The more instances that are created the more an orchestration platform like Kubernetes becomes a requirement... and the more pieces that need to communication [with the potential for issues] then the more something like Istio becomes useful for diagnostics.
yacovm (Thu, 23 May 2019 00:27:46 GMT):
> 'the peer/orderer processes are not microservices' ... they are also quite clearly not designed for high availability nor cloud environments either.
why are they not designed for HA ? can you give an example?
The latest v1.4.1 comes with a Raft orderer, and peers monitor each other and can tell the SDK (via service discovery) which peers are alive and which peers are not, etc.
iramiller (Thu, 23 May 2019 14:22:57 GMT):
@yacovm by focusing on the specific instance being connected to by name (i.e. peer-1.org) instead of a common endpoint (i.e. peer.org) the high availability and load balancing concepts of cloud platforms break down. The difficulties being discussed in the 'cloud native' FAB-13582 design document for load balancing and additionally the issues raised for a implementing Raft with external TLS management stem from a fundamental design issue. If these concepts were imple
iramiller (Thu, 23 May 2019 14:22:57 GMT):
@yacovm by focusing on the specific instance being connected to by name (i.e. peer-1.org) instead of a common endpoint (i.e. peer.org) the high availability and load balancing concepts of cloud platforms break down. The difficulties being discussed in the 'cloud native' FAB-13582 design document for load balancing and additionally the issues raised for a implementing Raft with external TLS management stem from a fundamental design issue. If these concepts were implemented with high availability in mind the transport layer would not be the defining identity, the signature on the content would be instead
iramiller (Thu, 23 May 2019 14:26:14 GMT):
A payload signed by an identity is equally valid when delivered by any common transport (secured or not -- although I am absolutely not advocating abandoning TLS). Proper isolation of the concerns of transport security from the blockchain functional operations would greatly simplify cloud deployment while providing additional flexibility.
yacovm (Thu, 23 May 2019 15:18:45 GMT):
> by focusing on the specific instance being connected to by name (i.e. peer-1.org) instead of a common endpoint (i.e. peer.org) the high availability and load balancing concepts of cloud platforms break down.
So that's exactly why I think the cloud approach of having a common endpoint that abstracts out the "backend" endopoint is simply wrong for a permissioned Blockchain platform like Fabric.
If everything is hosted in your home cloud environment - the cloud approach you describe works very well.
However - the endpoint can belong to a different organization than yourself, and when that endpoint "changes location" - that organization is not going to update "your cloud" about its new location, right? Now, if you have a small amount of organiztions - 2 or 3, it might be useful to model the peers as logical endpoints and not have them mimic real servers (such as Fabric does), however - as the number of parties in the channel grows - the advantage for this diminishes.
yacovm (Thu, 23 May 2019 15:19:33 GMT):
You're basically asking - why does Fabric treat endpoints as servers, and not as resources like everywhere in the modern world
yacovm (Thu, 23 May 2019 15:20:00 GMT):
so first of all - the orderer endpoints that are client facing can be modeled like this
iramiller (Thu, 23 May 2019 15:20:33 GMT):
peer.domain.com doesn't change ... the signature of blocks delivered should be `peer of member` ... that doesn't change ... the instance doing the work (one or more) is what can change ... that detail does not matter to a calling member.
yacovm (Thu, 23 May 2019 15:20:33 GMT):
but in Raft - that's not going to work, because we require mutual TLS between endpoints, because every replica "counts"
yacovm (Thu, 23 May 2019 15:21:32 GMT):
> .. the signature of blocks delivered should be peer of member ... that doesn't change ... the instance doing the work (one or more) is what can change ... that detail does not matter to a calling member.
There is more to Fabric node authentication than signatures and verification.
yacovm (Thu, 23 May 2019 15:21:32 GMT):
> .. the signature of blocks delivered should be peer of member ... that doesn't change ... the instance doing the work (one or more) is what can change ... that detail does not matter to a calling member.
There is more to Fabric node authentication than signatures and verification of blocks.
yacovm (Thu, 23 May 2019 15:22:19 GMT):
For instance, gossip enforces mutual TLS because Fabric is x509 independent, but we use mutual TLS. That means, that to identify a peer you have to bind the TLS certificate to the peer's non x509 (in theory, right?) identity
yacovm (Thu, 23 May 2019 15:24:47 GMT):
I agree, btw - that the chaincode deployment and runtime architecture can improve
iramiller (Thu, 23 May 2019 15:24:50 GMT):
Raft is a different animal by its very nature ... in a cloud of course instances need to be free to come and go and in fact that is the essence of the algorithm. Making that work at all was a hard job and kudos to the team for getting that done.
yacovm (Thu, 23 May 2019 15:25:24 GMT):
I actually think (well, humble brag) that we did a good job with making Raft easy to manage
iramiller (Thu, 23 May 2019 15:25:29 GMT):
I don't know that there would be an elegant solution in the cloud environments for that that still provides the guarantees a blockchain needs
yacovm (Thu, 23 May 2019 15:25:29 GMT):
there are no API calls for Raft
yacovm (Thu, 23 May 2019 15:25:39 GMT):
it just "knows" what it needs to replicate, and replicates it
yacovm (Thu, 23 May 2019 15:26:15 GMT):
each replica figures out which channels it should join, and replicates them when it joins the cluster as a fresh node
iramiller (Thu, 23 May 2019 15:26:35 GMT):
The ideal of a cloud environment is no special specific instances ... any instance can come and go at any time... that doesn't fit well with any solution that targets a specific instance
yacovm (Thu, 23 May 2019 15:27:03 GMT):
well, you know that classical consensus algorithms dictate certain conditions.
yacovm (Thu, 23 May 2019 15:27:37 GMT):
one of them, is that a replica has state and an identity and cannot be replaced by another identity.
yacovm (Thu, 23 May 2019 15:27:46 GMT):
this has very severe implications for DR
yacovm (Thu, 23 May 2019 15:28:04 GMT):
if you run Raft you better have 3 or more availability zones, or a backup of some of the private keys
yacovm (Thu, 23 May 2019 15:28:25 GMT):
because if half of the nodes are destroyed forever, the cluster is dead
iramiller (Thu, 23 May 2019 15:29:10 GMT):
the classic example I use locally is that a blockchain is super secure when it is running ... and make sure you doing 'lock your keys in your car' or it is a very secure and useless large object
yacovm (Thu, 23 May 2019 15:29:29 GMT):
and, by all means if you think we have room for improvement in the architecture side - open a JIRA, tag some people like me and @sykesm and we can start a discussion.
sykesm (Thu, 23 May 2019 15:29:29 GMT):
Has joined the channel.
yacovm (Thu, 23 May 2019 15:30:01 GMT):
i think you have good ideas and insights and it's a shame they are all buried in a channel that almost no one reads ;)
iramiller (Thu, 23 May 2019 15:30:38 GMT):
I do in fact contribute to the existing JIRA issues targeting these areas ... and I also raise these concerns in these channels as through the discussion we have here (and the countless experts that are watching) ... better solutions can come out
yacovm (Thu, 23 May 2019 15:30:56 GMT):
oh, ok
iramiller (Thu, 23 May 2019 15:34:03 GMT):
it is sad that some of the history scrolls away to be forgotten ... but the increased understanding by those that do read remains ... and from a personal point of view it informs my future comments and contributions here and in those JIRA tickets mentioned. While an ideal might be contributing directly to the source code that has presented some challenges given my current situation ... however contributing to these discussions is much easier to do. Supporting other members directly through DM here has also been a way to give back to this community.
iramiller (Thu, 23 May 2019 15:34:58 GMT):
Also worth pointing out is that I try and always make the point that there are competing use cases here and the needs of those running in these Kubernetes environments are not the only ones (or even the most important). Developing a large system is a set of compromises.
yacovm (Thu, 23 May 2019 15:35:19 GMT):
yep good point
Ramrockez143 (Sun, 26 May 2019 06:57:41 GMT):
how we can deploy kafka cluster across multiple eks clusters
Ramrockez143 (Sun, 26 May 2019 06:58:31 GMT):
we scenario like we need to deploy hypereledger across multiple clusters
raj_shekhar (Sun, 26 May 2019 09:48:30 GMT):
many like me are watching and enjoying ;) ,,,,, very insightful (Y)
ihormudryy (Tue, 28 May 2019 14:13:23 GMT):
health
iramiller (Tue, 28 May 2019 14:33:42 GMT):
You might want to look into Terraform ... we have been using this to handle our multi-cloud deployments. Shipping a Terraform script to a new org to bootstrap their cluster is great because there are so many steps required that a script is the only way to ensure things are done the same way every time... also good for quickly building/rebuilding test and development environments.
iramiller (Tue, 28 May 2019 14:36:26 GMT):
Related: microk8s has been fantastic for our development kubernetes environments. We spin up a large VM in the cloud and lay down a full cluster on it then turn the keys over to a developer. This gives them a very easy to reset environment for testing chaincode/channels/new applications without impacting others as would be the case if they were using our integration or QA networks.
AndresMartinezMelgar.itcl (Wed, 29 May 2019 06:18:34 GMT):
Anyone try to create a volume mount from / ??
I get to create one volume mount --> /HLF
RodrigoMedeiros (Wed, 29 May 2019 17:18:10 GMT):
Has joined the channel.
MattMilligan (Fri, 31 May 2019 16:09:49 GMT):
Has joined the channel.
hanubc7743 (Sat, 01 Jun 2019 11:58:10 GMT):
Has joined the channel.
Ramrockez143 (Tue, 04 Jun 2019 06:20:37 GMT):
how to deploy single zookeeper cluster across multiple eks cluster
Ramrockez143 (Tue, 04 Jun 2019 06:23:28 GMT):
when iam trying to deploy zookeeper 1 in cluster1,zookeeper2 in cluster2 and zookeeper3 cluster in cluster3,but leader election is not happening,if any body knew this scenario give me suggestion
vanitas92 (Tue, 04 Jun 2019 07:01:13 GMT):
@Ramrockez143 if you are deploying a new network from scratch i will suggest to deploy using the Raft consensus in version 1.4.1 as it is much simpler and specially designed for distributed consensus, as it is what you want to achieve in multiple cloud clusters
Ramrockez143 (Tue, 04 Jun 2019 08:18:04 GMT):
ok
Ramrockez143 (Tue, 04 Jun 2019 08:19:01 GMT):
yes we are deploying from the scratch
Chandoo (Tue, 04 Jun 2019 14:15:30 GMT):
Hi
Chandoo (Tue, 04 Jun 2019 14:16:09 GMT):
Did anyone tried deploying orderers with RAFT on multiple kubernetes clusters?
Chandoo (Tue, 04 Jun 2019 14:17:02 GMT):
I am trying it on EKS amazon managed service.
Chandoo (Tue, 04 Jun 2019 14:17:48 GMT):
if any one tried how the setup was done in reaching orderers on the other cluster
Chandoo (Tue, 04 Jun 2019 14:18:36 GMT):
I know there is a way to do with Federated Services, but i am trying to achieve with AWS EKS
vanitas92 (Fri, 07 Jun 2019 09:21:00 GMT):
Hello guys! Have someone tried to protect the peer or orderer endpoints by not exposing them directly on the internet when using multiple kubernetes instances? Like putting an nginx oir something in front of them? What have you used to deal with this? Thank you!
Kevin_Ko (Fri, 07 Jun 2019 16:10:55 GMT):
Has joined the channel.
raj_shekhar (Mon, 10 Jun 2019 10:53:21 GMT):
I have implemented it in GKE using Ingress ,,, worked fine ,,, but looking for more sophisticated ways to do same.
iramiller (Mon, 10 Jun 2019 16:05:20 GMT):
We use a wireguard VPN solution to make a private network of peer/order endpoints between the various orgs within our consortium. All addressing works as `peer-n.membername` (or `peer-n.membername.svc.cluster.local) across the entire distributed network (including GKE, AKS, EKS environments)
iramiller (Mon, 10 Jun 2019 16:05:20 GMT):
We use a wireguard VPN solution to make a private network of peer/order endpoints between the various orgs within our consortium. All addressing works as `peer-n.membername` (or `peer-n.membername.svc.cluster.local`) across the entire distributed network (including GKE, AKS, EKS environments)
AshishMishra 1 (Tue, 11 Jun 2019 06:14:50 GMT):
@iramiller in this private network using private hostnames, how do you validate your certs using a CA? or do you use the self signed certs of fabric CA.
AshishMishra 1 (Tue, 11 Jun 2019 06:15:46 GMT):
@raj_shekhar did you face any issue with URLs while using TLS?
mustafahusain (Tue, 11 Jun 2019 06:35:10 GMT):
Has joined the channel.
raj_shekhar (Tue, 11 Jun 2019 07:24:01 GMT):
@AshishMishra 1 I have set mutual TLS off as currently there is single cluster and exposing endpoints via TLS configured ingress.
AshishMishra 1 (Tue, 11 Jun 2019 07:25:36 GMT):
@raj_shekhar so are you terminating the TLS at the ingress or at the fabric peer?
raj_shekhar (Tue, 11 Jun 2019 07:27:45 GMT):
at the ingress.
AshishMishra 1 (Tue, 11 Jun 2019 07:32:08 GMT):
nice.. so all the endpoints in your configtx for fabric nodes/endpoints are your ingress endpoints or the internal service ones?
raj_shekhar (Tue, 11 Jun 2019 08:02:24 GMT):
nopes,
I have only exposed services for fabric-SDK, so that client application from outside can connect, everything else is same as of usual fabric network with TLS set as off,,, It is a single cluster network so issue in it.
raj_shekhar (Tue, 11 Jun 2019 08:02:24 GMT):
nopes,
I have only exposed services for fabric-SDK, so that client application from outside can connect, everything else is same as of usual fabric network with TLS set as off,,, It is a single cluster network so no issue in it.
tballast (Tue, 11 Jun 2019 13:18:13 GMT):
Has joined the channel.
iramiller (Tue, 11 Jun 2019 14:31:53 GMT):
We use our own self-signed root/intermediate CA.
iramiller (Tue, 11 Jun 2019 14:32:59 GMT):
We do not have issues with TLS (or mutual TLS) in our setup. The external facing APIs which provide business methods on top of the blockchain run as standard HTTP(S) web services and use a public CA root to pin their certificates
iramiller (Tue, 11 Jun 2019 14:41:37 GMT):
Our configtx (system-channel configuration) endpoints are all using internal DNS names. As a private blockchain network we have not needed to expose hyperledger endpoints directly to the internet.
Business processes that invoke chain code are exposed with custom REST APIs which are easy to handle with traditional Kubernetes orchestration. While a message/event based API would be really powerful and a closer match to what the blockchain is actually doing we have contained that complexity behind basic request/reply methods that are easier for a wider audience of developers to integrate with. The more specific needs of routing and load balancing against various peer and orderer endpoints are wrapped up in our own internal SDK which is shared out for members to use when connecting to Hyperledger to build their own APIs.
YassineAmor (Wed, 12 Jun 2019 10:00:39 GMT):
Has joined the channel.
raj_shekhar (Thu, 13 Jun 2019 04:35:58 GMT):
error.png
raj_shekhar (Thu, 13 Jun 2019 04:36:58 GMT):
I have set the policy in configtx file and after that it is coming. Any pointers on policies topic other than HL docs?
JayJong (Thu, 13 Jun 2019 07:59:54 GMT):
Hi all, if the peer pods in the kubernetes cluster is down, what is the best way to bring it up again with the same msp?
iramiller (Thu, 13 Jun 2019 16:24:18 GMT):
@JayJong ... not much to go on in your statement ... but I would recommend that your MSP files are stored in a secret which is mounted into the peer pod. This is also how you should handle configuration, either through extensive ENV vars mapped from a configmap or as the configfile mounted into the peer pod.
AndresMartinezMelgar.itcl (Sun, 16 Jun 2019 15:02:11 GMT):
hi anyone have tryied to create a https service inside kubernetes? i swap http to https. When i have https service it works but in https doesnrt
AndresMartinezMelgar.itcl (Sun, 16 Jun 2019 15:02:58 GMT):
hi anyone have tried to create a https service inside kubernetes? i swap http to https. When i have https service it works but in https doesn't
AndresMartinezMelgar.itcl (Sun, 16 Jun 2019 15:03:42 GMT):
I look that i have to make a ingress module with a secret, but i cant make it work
hanubc7743 (Sun, 16 Jun 2019 17:43:40 GMT):
Hi I am getting below error
./bin/configtxgen -profile OrdererGenesis -outputBlock ./channel-artifacts/genesis.block
2019-06-16 23:07:20.775 IST [common.tools.configtxgen] main -> WARN 001 Omitting the channel ID for configtxgen for output operations is deprecated. Explicitly passing the channel ID will be required in the future, defaulting to 'testchainid'.
2019-06-16 23:07:20.775 IST [common.tools.configtxgen] main -> INFO 002 Loading configuration
2019-06-16 23:07:20.788 IST [common.tools.configtxgen.localconfig] Load -> PANI 003 Error unmarshaling config into struct: 1 error(s) decoding:
* '' has invalid keys: v1_1
2019-06-16 23:07:20.788 IST [common.tools.configtxgen] func1 -> PANI 004 Error unmarshaling config into struct: 1 error(s) decoding:
* '' has invalid keys: v1_1
panic: Error unmarshaling config into struct: 1 error(s) decoding:
* '' has invalid keys: v1_1 [recovered]
panic: Error unmarshaling config into struct: 1 error(s) decoding:
* '' has invalid keys: v1_1
goroutine 1 [running]:
github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc000121970, 0x0, 0x0, 0x0)
/w/workspace/fabric-release-jobs-x86_64/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore/entry.go:229 +0x515
github.com/hyperledger/fabric/vendor/go.uber.org/zap.(*SugaredLogger).log(0xc0000b2240, 0xc0002f9804, 0xc00009b1a0, 0x59, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/w/workspace/fabric-release-jobs-x86_64/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:234 +0xf6
github.com/hyperledger/fabric/vendor/go.uber.org/zap.(*SugaredLogger).Panicf(0xc0000b2240, 0xc00009b1a0, 0x59, 0x0, 0x0, 0x0)
/w/workspace/fabric-release-jobs-x86_64/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:159 +0x79
github.com/hyperledger/fabric/common/flogging.(*FabricLogger).Panic(0xc0000b2248, 0xc0002f9938, 0x1, 0x1)
/w/workspace/fabric-release-jobs-x86_64/gopath/src/github.com/hyperledger/fabric/common/flogging/zap.go:73 +0x75
main.main.func1()
/w/workspace/fabric-release-jobs-x86_64/gopath/src/github.com/hyperledger/fabric/common/tools/configtxgen/main.go:250 +0x1a9
panic(0xd4fc40, 0xc0003018e0)
/opt/go/go1.11.1.linux.amd64/src/runtime/panic.go:513 +0x1b9
github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc000121970, 0x0, 0x0, 0x0)
/w/workspace/fabric-release-jobs-x86_64/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore/entry.go:229 +0x515
github.com/hyperledger/fabric/vendor/go.uber.org/zap.(*SugaredLogger).log(0xc0000b2220, 0xc0002f9c04, 0xc00009b080, 0x59, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/w/workspace/fabric-release-jobs-x86_64/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:234 +0xf6
github.com/hyperledger/fabric/vendor/go.uber.org/zap.(*SugaredLogger).Panicf(0xc0000b2220, 0xc00009b080, 0x59, 0x0, 0x0, 0x0)
/w/workspace/fabric-release-jobs-x86_64/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:159 +0x79
github.com/hyperledger/fabric/common/flogging.(*FabricLogger).Panic(0xc0000b2228, 0xc0002f9d98, 0x2, 0x2)
/w/workspace/fabric-release-jobs-x86_64/gopath/src/github.com/hyperledger/fabric/common/flogging/zap.go:73 +0x75
github.com/hyperledger/fabric/common/tools/configtxgen/localconfig.Load(0x7ffeb116516d, 0xe, 0x0, 0x0, 0x0, 0x1)
/w/workspace/fabric-release-jobs-x86_64/gopath/src/github.com/hyperledger/fabric/common/tools/configtxgen/localconfig/config.go:282 +0x580
main.main()
/w/workspace/fabric-release-jobs-x86_64/gopath/src/github.com/hyperledger/fabric/common/tools/configtxgen/main.go:261 +0xb2f
Anyone know any solution please?
raj_shekhar (Mon, 17 Jun 2019 07:06:38 GMT):
Can you plz share your configtx.yaml file.
raj_shekhar (Mon, 17 Jun 2019 07:19:41 GMT):
To enable ssl using ingress you can do below.
1. Get cert and keys from the letsencrypt (open source and free)
2. create secret of cert in K8s - kubectl create secret tls ${CERT_NAME} --key ${KEY_FILE} --cert ${CERT_FILE}
3. Configure the ingress with hosts and secret like below
tls:
- hosts:
- app.example.com
secretName: cert-secret
hanubc7743 (Mon, 17 Jun 2019 07:40:59 GMT):
------
Organizations:
- &OrdererOrg
Name: OrdererOrg
ID: OrdererMSP
MSPDir: crypto-config/ordererOrganizations/example.com/msp
- &Org1
Name: Org1MSP
ID: Org1MSP
MSPDir: crypto-config/peerOrganizations/org1.example.com/msp
AnchorPeers:
- Host: peer0.org1.example.com
Port: 7051
- &Org2
Name: Org2MSP
ID: Org2MSP
MSPDir: crypto-config/peerOrganizations/org2.example.com/msp
AnchorPeers:
- Host: peer0.org2.example.com
Port: 7051
- &Org3
Name: Org3MSP
ID: Org3MSP
MSPDir: crypto-config/peerOrganizations/org3.example.com/msp
AnchorPeers:
- Host: peer0.org3.example.com
Port: 7051
Orderer: &OrdererDefaults
OrdererType: solo
Addresses:
- orderer.example.com:7050
BatchTimeout: 2s
BatchSize:
MaxMessageCount: 10
AbsoluteMaxBytes: 99 MB
PreferredMaxBytes: 512 KB
Kafka:
Brokers:
- 127.0.0.1:9092
Organizations:
Application: &ApplicationDefaults
Organizations:
Capabilities:
Global: &ChannelCapabilities
V1_1: true
Orderer: &OrdererCapabilities
V1_1: true
Application: &ApplicationCapabilities
V1_1: true
Profiles:
OrdererGenesis:
Capabilities:
<<: *ChannelCapabilities
Orderer:
<<: *OrdererDefaults
Organizations:
- *OrdererOrg
Capabilities:
<<: *OrdererCapabilities
Consortiums:
SampleConsortium:
Organizations:
- *Org1
- *Org2
- *Org3
ChannelAll:
Consortium: SampleConsortium
Application:
<<: *ApplicationDefaults
Organizations:
- *Org1
- *Org2
- *Org3
Capabilities:
<<: *ApplicationCapabilities
raj_shekhar (Mon, 17 Jun 2019 08:26:32 GMT):
Can u try formatting your configtx file as per yaml format.... there are tools available online to do so.
hanubc7743 (Mon, 17 Jun 2019 08:31:10 GMT):
I have checked its valid
hanubc7743 (Mon, 17 Jun 2019 08:31:17 GMT):
Organizations:
- &OrdererOrg
Name: OrdererOrg
ID: OrdererMSP
MSPDir: crypto-config/ordererOrganizations/example.com/msp
- &Org1
Name: Org1MSP
ID: Org1MSP
MSPDir: crypto-config/peerOrganizations/org1.example.com/msp
AnchorPeers:
- Host: peer0.org1.example.com
Port: 7051
- &Org2
Name: Org2MSP
ID: Org2MSP
MSPDir: crypto-config/peerOrganizations/org2.example.com/msp
AnchorPeers:
- Host: peer0.org2.example.com
Port: 7051
- &Org3
Name: Org3MSP
ID: Org3MSP
MSPDir: crypto-config/peerOrganizations/org3.example.com/msp
AnchorPeers:
- Host: peer0.org3.example.com
Port: 7051
Orderer: &OrdererDefaults
OrdererType: solo
Addresses:
- orderer.example.com:7050
BatchTimeout: 2s
BatchSize:
MaxMessageCount: 10
AbsoluteMaxBytes: 99 MB
PreferredMaxBytes: 512 KB
Kafka:
Brokers:
- 127.0.0.1:9092
Organizations:
Application: &ApplicationDefaults
Organizations:
Capabilities:
Global: &ChannelCapabilities
V1_1: true
Orderer: &OrdererCapabilities
V1_1: true
Application: &ApplicationCapabilities
V1_1: true
Profiles:
OrdererGenesis:
Capabilities:
<<: *ChannelCapabilities
Orderer:
<<: *OrdererDefaults
Organizations:
- *OrdererOrg
Capabilities:
<<: *OrdererCapabilities
Consortiums:
SampleConsortium:
Organizations:
- *Org1
- *Org2
- *Org3
ChannelAll:
Consortium: SampleConsortium
Application:
<<: *ApplicationDefaults
Organizations:
- *Org1
- *Org2
- *Org3
Capabilities:
<<: *ApplicationCapabilities
aatkddny (Mon, 17 Jun 2019 13:34:49 GMT):
Does anyone have a yaml file for a raft orderer they'd be willing to share? I'm struggling to parse the changes vs the kafka setup and would dearly love to kickstart this process without a bunch of trial and error.
yacovm (Mon, 17 Jun 2019 14:20:55 GMT):
@aatkddny just look at the section of etcdraft in orderer.yaml and the cluster section in orderer.yaml
aatkddny (Mon, 17 Jun 2019 15:15:46 GMT):
@yacovm - I was referring to the actual k8s yaml definition file.
I've been looking at the config file you checked in on mar 23 in sampleconfig/orderer.yaml.
I need to relate that to a real k8s deployment file. The one I have for kafka externalizes all the overrides, rather than passing in the configuration (since I never figured out how to do that) and I was trying to avoid a trial and error time suck moving that to this. My kubernetes-fu is more of the "if stack overflow didn't work, google is your friend" variety here.
yacovm (Mon, 17 Jun 2019 15:16:12 GMT):
ah... no, don't have such a file, sorry.
aatkddny (Mon, 17 Jun 2019 15:16:51 GMT):
nm. appreciate the response.
kantipov (Mon, 17 Jun 2019 15:51:31 GMT):
Has joined the channel.
iramiller (Mon, 17 Jun 2019 17:32:35 GMT):
@aatkddny I am not exactly sure what kind of k8s file you are looking for ... but you can embed the yaml config files from hyperledger inside of a configmap yaml file using the standard indentation method for scope ... then map that configmap into the container as a file ... is that what you are trying to do?
aatkddny (Mon, 17 Jun 2019 18:29:11 GMT):
I wasn
aatkddny (Mon, 17 Jun 2019 18:29:11 GMT):
I wasn't - mainly because I was set up to override the defaults using viper(?).
Mostly because I hacked at my docker-compose files until they worked.
If there's a pattern to embed the config files directly I'd happily pivot to do that.
aatkddny (Mon, 17 Jun 2019 18:29:11 GMT):
I wasn't - mainly because I was set up to override the defaults using viper(?).
Mostly because I hacked at my docker-compose files until they worked.
If there's a pattern to embed the config files directly I'd happily pivot to do that. It's likely to be simpler going forward.
aatkddny (Mon, 17 Jun 2019 18:29:11 GMT):
I wasn't - mainly because I was set up to override the defaults using viper(?).
Mostly because I hacked at my docker-compose files until they worked with k8s. Learned a bit about containerization, but clearly not enough for this job.
If there's a pattern to embed the config files directly I'd happily pivot to do that. It's likely to be simpler going forward.
aatkddny (Mon, 17 Jun 2019 18:35:30 GMT):
So just to clarify - this is one of my local orderer configs
```
- name: orderer0
image: hyperledger/fabric-orderer
command:
- sh
- -c
- orderer
env:
- name: GRPC_TRACE
value: all=true
- name: GRPC_VERBOSITY
value: debug
- name: ORDERER_CFG_PATH
value: /share
- name: ORDERER_GENERAL_LEDGERTYPE
value: file
- name: ORDERER_FILELEDGER_LOCATION
value: /share/ledger/orderer0
- name: ORDERER_GENERAL_BATCHTIMEOUT
value: 1s
- name: ORDERER_GENERAL_BATCHSIZE_MAXMESSAGECOUNT
value: "50"
- name: ORDERER_GENERAL_MAXWINDOWSIZE
value: "1000"
- name: ORDERER_GENERAL_LISTENADDRESS
value: 0.0.0.0
- name: ORDERER_GENERAL_LISTENPORT
value: "7050"
- name: ORDERER_GENERAL_LOGLEVEL
value: DEBUG
- name: ORDERER_GENERAL_LOCALMSPID
value: OrdererMSP
- name: ORDERER_GENERAL_LOCALMSPDIR
value: /share/generated/crypto-config/ordererOrganizations/orderer/orderers/orderer0.orderer/msp
- name: ORDERER_GENERAL_GENESISMETHOD
value: file
- name: ORDERER_GENERAL_GENESISFILE
value: /share/generated/channel-artifacts/genesis.block
- name: ORDERER_GENERAL_GENESISPROFILE
value: Genesis
- name: ORDERER_GENERAL_TLS_ENABLED
value: "false"
- name: GODEBUG
value: netdns=go
volumeMounts:
- mountPath: /share
name: om
restartPolicy: Always
volumes:
- name: om
persistentVolumeClaim:
claimName: bc-storage
```
iramiller (Mon, 17 Jun 2019 18:35:37 GMT):
in your deployment use
```
volumeMounts:
- mountPath: /etc/hyperledger/peer
name: my-config-file
```
then make your config files roughly like this helm example
```
apiVersion: v1
kind: ConfigMap
metadata:
name: my-config-file
data:
core.yaml: |
# Copyright IBM Corp. All Rights Reserved.
#
# SPDX-License-Identifier: Apache-2.0
# [rest of config file follows with 4 space indent]
```
iramiller (Mon, 17 Jun 2019 18:36:39 GMT):
the issue we found with relying on the ENV configs is mostly due to the fact that settings are not discoverable nor are there any consistency checks for misspelled entries
iramiller (Mon, 17 Jun 2019 18:37:21 GMT):
so we put a bunch of work into creating config files set the way we wanted so that a platform default wouldn't accidentally step in and break our production or test environments
aatkddny (Mon, 17 Jun 2019 18:37:34 GMT):
I autogen all mine from an application I wrote. We have quite a number of these things and management was an issue.
iramiller (Mon, 17 Jun 2019 18:37:54 GMT):
of course all of our deployments are scripted/managed
iramiller (Mon, 17 Jun 2019 18:38:08 GMT):
but it is still very nice for auditing to have everything in the config file consistent
aatkddny (Mon, 17 Jun 2019 18:39:26 GMT):
You are way ahead of us in that department.
I do have code to generate the scripts, but here the problem is I need to change the templating every time there's a new version and that's where the rub is.
I'm struggling to figure out where to start in this case.
aatkddny (Mon, 17 Jun 2019 18:39:26 GMT):
You are way ahead of us in that department.
I do have code to generate the scripts, but here the problem is I need to change the templating every time there's a new version and that's where the rub is.
I'm struggling to figure out where to start in this case. It's probably very simple if I can find a -compose version to hack at but even that I'm finding tough to find. Too new.
iramiller (Mon, 17 Jun 2019 18:42:03 GMT):
honestly we lost quite a bit of ground building on the docker-compose and script file approach a year ago ... eventually I did a parallel build up of helm charts and HashiCorp Terraform scripting for the major components (peer, orderer, etc) and when test was working we did a full cut over
iramiller (Mon, 17 Jun 2019 18:43:10 GMT):
so many headaches just getting a clean repeatable deployment prior to investing in that process. it isn't that it can't be done its just that the Kubernetes environment for production is quite a bit different use case than the proof of concept docker-compose systems
iramiller (Mon, 17 Jun 2019 18:44:06 GMT):
the unfortunate part of what we have done is that our scripts and charts are highly specific to our environments and not very useful for the community at large ...
iramiller (Mon, 17 Jun 2019 18:44:49 GMT):
but I am happy to share pieces or experience to those doing something similar ... as a startup there just isn't time with how fast we are moving to build up a good generic system ...
aatkddny (Mon, 17 Jun 2019 18:54:01 GMT):
I may have not been as clear as I might have been. I used the docker-compose examples to give me the parameters I needed to build templates for my kubernetes setups. my app generates the k8s yaml files for each of the orgs from these - so i can deploy everything with kubectl - and a set of bash scripts that i can run to do this in 3 stages - i found i needed to let the mysql ca db(s) initialize because i couldn't get them inside the same pod as the ca and i needed to let kafka settle down before starting the orderers, so it required 3 scripts in all. We don't use a lot of containerization in house, and we are just starting using helm a year or so after this was developed, so it is all a bit of a hack.
If I ever get time I'll do proper helm templates for it all, but right now I just need to get orderers with raft up and running.
iramiller (Mon, 17 Jun 2019 19:02:21 GMT):
we are still not using Raft yet internally for various reasons ... probably similar story to why you haven't built up helm charts for everything .. broken stuff gets first priority followed by features that make money ... refinements are always down the list until something makes them critical...
iramiller (Mon, 17 Jun 2019 19:03:19 GMT):
from what I saw of the orderer piece with regards to raft it will require a completely new deployment setup that doesn't use much of our existing orderer deployment setup...
iramiller (Mon, 17 Jun 2019 19:04:25 GMT):
the pinned TLS and blockchain updates to make changes will require a pretty significant investment in scripting/integration to adopt as our TLS certificates have typically rotated on a more frequent basis.
iramiller (Mon, 17 Jun 2019 19:06:03 GMT):
when we finally get around to fully working through the raft upgrade process I am sure I will know more. At the moment the CA challenges you mentioned our of a greater concern to us and we have been working to adopt HashiCorp Vault instead of fabric_ca as it's feature set makes the security team happy and the many cloud environment backing stores make DevOps happy.
vanitas92 (Mon, 17 Jun 2019 21:08:41 GMT):
@aatkddny and anyone who is interested, find here my config using a RAFT cluster in Kubernetes that is working. Apart from setting the genesis block to etcdraft consensus and node names, the config for each orderer is as follows:
```yaml
env:
- name: FABRIC_LOGGING_SPEC
value: INFO
- name: ORDERER_OPERATIONS_LISTENADDRESS
value: 0.0.0.0:8443
- name: ORDERER_METRICS_PROVIDER
value: prometheus
- name: ORDERER_GENERAL_GENESISFILE
value: /var/hyperledger/orderer/genesis.block
- name: ORDERER_GENERAL_GENESISMETHOD
value: file
- name: ORDERER_GENERAL_LISTENADDRESS
value: 0.0.0.0
- name: ORDERER_GENERAL_LOCALMSPDIR
value: /var/hyperledger/orderer/msp
- name: ORDERER_GENERAL_LOCALMSPID
value: OrdererMSP
- name: ORDERER_GENERAL_TLS_CERTIFICATE
value: /var/hyperledger/orderer/tls/server.crt
- name: ORDERER_GENERAL_TLS_ENABLED
value: "true"
- name: ORDERER_GENERAL_TLS_PRIVATEKEY
value: /var/hyperledger/orderer/tls/server.key
- name: ORDERER_GENERAL_TLS_ROOTCAS
value: '[/var/hyperledger/orderer/tls/ca.crt]'
- name: ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE
value: /var/hyperledger/orderer/tls/server.crt
- name: ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY
value: /var/hyperledger/orderer/tls/server.key
- name: ORDERER_GENERAL_CLUSTER_ROOTCAS
value: '[/var/hyperledger/orderer/tls/ca.crt]'
image: hyperledger/fabric-orderer:amd64-1.4.1
```
vanitas92 (Mon, 17 Jun 2019 21:09:51 GMT):
The most important setting is the last three options: `ORDERER_GENERAL_CLUSTER_*`, they enable the certificates necesary to establish a RAFT cluster
vanitas92 (Mon, 17 Jun 2019 21:10:21 GMT):
Is the same certificates if you enable TLS in your network
AshishMishra 1 (Mon, 17 Jun 2019 22:32:00 GMT):
@iramiller so basically none of your fabric end-points are exposed to public. So in case of a multi-org scenario where the other party is hosting their own fabric can't communicate with your fabric n/w unless they use a VPN kind of thing?
AshishMishra 1 (Mon, 17 Jun 2019 22:35:30 GMT):
@vanitas92 did you try something like envoy or linkerd ? I did a POC about a year ago to proxy the grpc traffic transparently to the peer node. It worked but created a lot of overhead to maintain it, so dropped it. Sadly I don't have any configuration with me.
iramiller (Mon, 17 Jun 2019 22:50:11 GMT):
@AshishMishra 1 ... as a private blockchain consortium there are many steps that would be required before someone could participate (including the generation of certificates). The setup of a secure wireguard tunnel actually simplifies the integration story between some of our members as approving internet access to these endpoints is a more complicated process than the private network config.
aatkddny (Mon, 17 Jun 2019 23:58:19 GMT):
@vanitas92 Thanks for that. Let me see if that gets me any further forward.
hanubc7743 (Tue, 18 Jun 2019 19:52:22 GMT):
Hi Anybody know how to work with hyperledger fabric java sdk and ipfs?
BrajeshKumar (Wed, 19 Jun 2019 09:00:29 GMT):
Has joined the channel.
cbf (Wed, 19 Jun 2019 15:39:46 GMT):
I'm unaware of any integrations of ipfs with Fabric, but there is a #fabric-sdk-java channel that might help yuo with the former
hanubc7743 (Wed, 19 Jun 2019 17:48:10 GMT):
Hi Can we integrate oracle db instead of couch db in hyperledger fabric?
iramiller (Wed, 19 Jun 2019 19:36:03 GMT):
@hanubc7743 maybe ask in the #fabric-peer-endorser-committer channel as that doesn't have anything to do with kubernetes. Before you ask though you might take a look at https://github.com/hyperledger/fabric/blob/release-1.4/core/ledger/util/couchdb/couchdb.go and think about what it would take to write your own interface to Oracle
iamksseo (Fri, 21 Jun 2019 08:00:48 GMT):
Thanks for that.
vanitas92 (Mon, 24 Jun 2019 09:43:16 GMT):
My case completely relies on internet since each organization is hosting their own infraestructure on premises or public clouds so need to protect the endpoints somehow. We are already using TLS over grpc but the endpoint is directly exposed to the internet and this concerns me.
vanitas92 (Mon, 24 Jun 2019 09:44:52 GMT):
@AshishMishra 1 so with envoy is possible to proxy to peers? how does it handle with TLS over grpc? Any remarkable experience? Thanks guys!
AshishMishra 1 (Mon, 24 Jun 2019 18:14:38 GMT):
@vanitas92 yes, I did make that work with envoy but it was a one to one mapping. So for each peer you need to have one envoy running. If I remember correctly I did use some SNI or some hostname overwriting to make it transparent to envoy. I wish I had the configurations with me. :|
AshishMishra 1 (Mon, 24 Jun 2019 18:17:38 GMT):
@iramiller yes, what you are saying is absolutely correct but what I am looking to achieve is more of a C2C model and not a B2B, so having a vpn or firwall based rules would be an overkill. What I wanted is users to freely run peers exposed to internet but via a proxy which would provide an additional layer of protection to prevent against attacks such as DDOS or TCP flood etc.
AshishMishra 1 (Mon, 24 Jun 2019 18:20:44 GMT):
@iramiller how is the vault working for you? I was thinking also to adopt something like vault which is more secure than k8s secrets which I am using right now. Also did that need any changes in the fabric code? thanks.
iramiller (Mon, 24 Jun 2019 18:28:18 GMT):
@AshishMishra 1 the wireguard vpn connection works similar to what you describe ... an internet facing port and traffic secured by preshared EC keys. I use a NAT style methodology implemented with IPTables to route traffic avoiding any Layer 7 incompatibilities. Ideally this entire configuration would be wrapped up in a custom Kubernetes ingress controller... I have not done this though nor have I looked for an implementation of it recently.
iramiller (Mon, 24 Jun 2019 18:30:49 GMT):
back when I designed our network architecture I was very surprised that no one had implemented a wireguard ingress controller (that I could find). The expedient method for implantation in a cross cloud way was using a small VM that was dual homed on the pod and external networks.
iramiller (Mon, 24 Jun 2019 18:30:49 GMT):
back when I designed our network architecture I was very surprised that no one had implemented a wireguard ingress controller (that I could find). The expedient method for implementation in a cross cloud way was using a small VM that was dual homed on the pod and external networks.
AshishMishra 1 (Mon, 24 Jun 2019 18:37:32 GMT):
Thanks, that's insightful
vanitas92 (Mon, 24 Jun 2019 20:24:08 GMT):
@AshishMishra 1 ok thanks! That is exactly what i want to achieve, kinda like a microservice architechture so its okay to have one to one mapping. At least would be some improvement for now. The certificates are the same in both peer and envoy server in front of him?
AshishMishra 1 (Mon, 24 Jun 2019 20:54:33 GMT):
@vanitas92 I think I used the tcp_proxy filter in envoy which acts as a pass through for tls traffic, so envoy doesn't have to do anything for the TLS. It's been a long time and I barely remember anything. Maybe some with a good experience with Envoy will be able to help better.
richardmurillo (Mon, 24 Jun 2019 21:00:28 GMT):
Has joined the channel.
MartinKanala (Tue, 25 Jun 2019 22:31:56 GMT):
Has joined the channel.
Ramrockez143 (Wed, 26 Jun 2019 06:14:38 GMT):
Hi, any body deployed zookeeper cluster across multiple kubernetes clusters,if any body knew this scenario kindly help me
aatkddny (Thu, 27 Jun 2019 15:21:44 GMT):
Is there a dockerized version of configtxlator anywhere?
Want to stick it in my cluster.
I know I can run it server local or from the command line, but I'd prefer to keep everything together without going to the trouble of rolling the whole thing myself.
Google wasn't my friend with a quick search for one.
UnaiUrkiaga (Fri, 28 Jun 2019 10:31:23 GMT):
Has joined the channel.
aatkddny (Fri, 28 Jun 2019 15:15:22 GMT):
@vanitas92 How did you get round the orderer TCP issue? I've set SANs but it's defaulting to IP addresses.
harsh-98 (Sat, 29 Jun 2019 09:28:12 GMT):
Has joined the channel.
harsh-98 (Sat, 29 Jun 2019 09:32:15 GMT):
I am running fabric on minikube. I have successfully created the channel and made peers join the channel and installed the chaincode. Instantiate chaincode fails. I know that it is related to chaincode container being created by peer not being recognized by kubernetes. Having this issue for 3-4 days not able to debug.
harsh-98 (Sat, 29 Jun 2019 09:33:03 GMT):
cli.png
harsh-98 (Sat, 29 Jun 2019 09:33:24 GMT):
peer.png
harsh-98 (Sat, 29 Jun 2019 09:37:08 GMT):
also where to put `DOCKER_OPTS="--dns=10.96.0.10 --dns=192.168.0.1 --dns-search default.svc.cluster.local --dns-search svc.cluster.local --dns-opt ndots:2 --dns-opt timeout:2 --dns-opt attempts:2 "` i mean in pod or host and any help,
davidkel (Sat, 29 Jun 2019 19:16:57 GMT):
Has joined the channel.
vanitas92 (Mon, 01 Jul 2019 07:24:44 GMT):
@aatkddny I did not experience any orderer tcp issue while setting up RAFT. What was the problem?
aatkddny (Mon, 01 Jul 2019 12:26:44 GMT):
The problem for those following along was that I was trying to put my orderers in with the peers - I have a consortium requirement.
There's a mismatch in the configtx org definition when you put the peerOrganization msp in the MSPDir profile in the org because you have peers but then have it use ordererOrganization certs to generate the genesis block because you want it to have orderers too.
iramiller (Mon, 01 Jul 2019 21:22:40 GMT):
configtxlater is in the `us.gcr.io/provenance-io/hyperledger-fabric-tools` image ... you should be able to start and run an instance using that image ... or just run it locally via shell/shell scripts as needed ... at least that has been our approach.
vanitas92 (Tue, 02 Jul 2019 08:42:14 GMT):
hey
vanitas92 (Tue, 02 Jul 2019 08:46:08 GMT):
guys i have managed to put an envoy in front of peer endpoint in order to protect it with TLS, here you can find the config file in case you need it:
```
static_resources:
listeners:
- name: listener_grpcs
address:
socket_address: { address: 0.0.0.0, port_value: 7051 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: backend
domains: ["*"]
routes:
- match:
prefix: "/"
grpc: {}
route:
cluster: service_grpcs
http_filters:
- name: envoy.router
typed_config: {}
tls_context:
common_tls_context:
tls_certificates:
- certificate_chain:
filename: "/etc/envoy/certs/server.crt"
private_key:
filename: "/etc/envoy/certs/server.key"
clusters:
- name: service_grpcs
connect_timeout: 5s
type: strict_dns
lb_policy: round_robin
http2_protocol_options: {}
load_assignment:
cluster_name: service_grpcs
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: peer0-org1
port_value: 7051
tls_context:
common_tls_context:
tls_certificates:
- certificate_chain:
filename: "/etc/envoy/certs/server.crt"
private_key:
filename: "/etc/envoy/certs/server.key"
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address:
address: 0.0.0.0
port_value: 8001
```
vanitas92 (Tue, 02 Jul 2019 08:46:08 GMT):
guys i have managed to put an envoy in front of peer endpoint in order to protect it with TLS, here you can find the config file in case you need it:
```
static_resources:
listeners:
- name: listener_grpcs
address:
socket_address: { address: 0.0.0.0, port_value: 7051 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: backend
domains: ["*"]
routes:
- match:
prefix: "/"
grpc: {}
route:
cluster: service_grpcs
http_filters:
- name: envoy.router
typed_config: {}
tls_context:
common_tls_context:
tls_certificates:
- certificate_chain:
filename: "/etc/envoy/certs/server.crt"
private_key:
filename: "/etc/envoy/certs/server.key"
clusters:
- name: service_grpcs
connect_timeout: 5s
type: strict_dns
lb_policy: round_robin
http2_protocol_options: {}
load_assignment:
cluster_name: service_grpcs
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: peer0-org1
port_value: 7051
tls_context:
common_tls_context:
tls_certificates:
- certificate_chain:
filename: "/etc/envoy/certs/server.crt"
private_key:
filename: "/etc/envoy/certs/server.key"
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address:
address: 0.0.0.0
port_value: 8001
```
pankajcheema (Tue, 02 Jul 2019 08:46:42 GMT):
https://chat.hyperledger.org/channel/fabric-questions?msg=sg9R8nTAD8tJtvixG
raj_shekhar (Thu, 04 Jul 2019 06:54:49 GMT):
Hi pankaj,
you are using peer cli to perform these operations? Just check the PEER MSP ID tag there ?
raj_shekhar (Thu, 04 Jul 2019 06:54:49 GMT):
Hi pankaj,
you are using peer cli to perform these operations? Just check the PEER MSP ID tag there
raj_shekhar (Thu, 04 Jul 2019 06:54:49 GMT):
Hi pankaj,
you are using peer cli to perform these operations? Just check the PEER MSP ID tag there .
pankajcheema (Thu, 04 Jul 2019 14:47:48 GMT):
@raj_shekhar yes I am aware with that
pankajcheema (Thu, 04 Jul 2019 14:48:03 GMT):
I was having some issue with my volume
pankajcheema (Thu, 04 Jul 2019 14:48:22 GMT):
Recreated them and it works fine
delao (Thu, 04 Jul 2019 16:59:11 GMT):
Has joined the channel.
raj_shekhar (Fri, 05 Jul 2019 04:05:15 GMT):
great...
raj_shekhar (Fri, 05 Jul 2019 04:05:55 GMT):
great,,
pankajcheema (Fri, 05 Jul 2019 09:59:29 GMT):
Hi All , I am facing a strange issue . I am having 2 ORGs each having 1 peer
pankajcheema (Fri, 05 Jul 2019 09:59:29 GMT):
Hi All , I am facing a strange issue . I am having 2 ORGs each having 1 peer . 1 orderer , my peer0 of org1 is able to communicate with ordering service using kubernetes service name `ordering-service` . but peer0 of org2 is throwing error like ``` grpc: addrConn.createTransport failed to connect to {ordering-service:7050 0
pankajcheema (Fri, 05 Jul 2019 10:01:57 GMT):
any idea ?
pankajcheema (Fri, 05 Jul 2019 10:02:26 GMT):
what i am missing ?
Dhiraj1990 (Mon, 08 Jul 2019 12:24:23 GMT):
Hello all,
I have deployed a network in k8s.Now i am able to install and instantiate the chaincode. Chaincode container also getting created after instantiation. But when i run commond
peer chaincode list --instantiated -C mychannel
Then
Dhiraj1990 (Mon, 08 Jul 2019 12:24:23 GMT):
Hello all,
I have deployed a network in k8s.Now i am able to install and instantiate the chaincode. Chaincode container also getting created after instantiation. But when i run commond
peer chaincode list --instantiated -C mychannel
Then i don't get instantiated chaincode in list. Please help me why this issue coming
aatkddny (Wed, 10 Jul 2019 00:20:47 GMT):
I'm not sure it's the correct answer, but I got round this by having my crypto config files look something like this:
```
- Hostname: raft0
CommonName: "{{.Hostname}}-{{.Domain}}"
SANS:
- "{{.Hostname}}"
- "*.{{.Domain}}"
- "localhost"
- "127.0.0.1"
```
aatkddny (Wed, 10 Jul 2019 00:20:47 GMT):
I'm not sure it's the official correct answer, but I got round this by having my crypto config files look something like this:
```
- Hostname: raft0
CommonName: "{{.Hostname}}-{{.Domain}}"
SANS:
- "{{.Hostname}}"
- "*.{{.Domain}}"
- "localhost"
- "127.0.0.1"
```
aatkddny (Thu, 11 Jul 2019 00:08:53 GMT):
Has anyone managed to get the java sdk to talk to a raft orderer inside k8s up to the point where you create a channel (which I can do) and then install and instantiate chaincode (which doesn't work so well)?
Or am I the first - which explains why I keep running across problems.
aatkddny (Thu, 11 Jul 2019 00:08:53 GMT):
Has anyone managed to get the java sdk to talk to a raft orderer inside k8s up to the point where you create a channel (which I can do) and then install and instantiate chaincode (which doesn't work so well)?
Or am I the first - which explains why I keep running across problems. The latest is to do with service discovery and eventing. It looks like somehow it switches to ip addresses from names internally.
harsh-98 (Thu, 11 Jul 2019 13:46:27 GMT):
I am also facing the same issue, Error: `Error: endorsement failure during query. response: status:500 message:"make sure the chaincode mycc has been successfully instantiated and try again: chaincode mycc not
found"
`
harsh-98 (Thu, 11 Jul 2019 13:46:27 GMT):
I am also facing the same issue, Error:
`Error: endorsement failure during query. response: status:500 message:"make sure the chaincode mycc has been successfully instantiated and try again: chaincode mycc not
found"`
harsh-98 (Thu, 11 Jul 2019 13:46:27 GMT):
I am also facing the same issue, Error:
`Error: could not assemble transaction, err proposal response was not successful, error code 500, msg chaincode registration failed: container exited with 0`
yacovm (Thu, 11 Jul 2019 20:43:18 GMT):
@aatkddny can you elaborate on SD and ip addresses?
aatkddny (Thu, 11 Jul 2019 22:31:58 GMT):
@yacovm rather than rehash this latest issue i'll link to it if that's ok.
https://chat.hyperledger.org/channel/fabric-sdk-java?msg=uZPd9ZxoKirq6aK4B
aatkddny (Thu, 11 Jul 2019 22:31:58 GMT):
@yacovm rather than rehash this latest issue i'll link to it if that's ok.
https://chat.hyperledger.org/channel/fabric-sdk-java?msg=uZPd9ZxoKirq6aK4B
I'm referring in particular to the second post in the thread - what happens after one gets past this problem by editing the hosts file.
aatkddny (Thu, 11 Jul 2019 22:31:58 GMT):
@yacovm rather than rehash this latest issue i'll link to it if that's ok.
https://chat.hyperledger.org/channel/fabric-sdk-java?msg=uZPd9ZxoKirq6aK4B
I'm referring in particular to the second post in the thread - what happens after one gets past this problem by editing the hosts file.
The ip address it is unhappy with is the one kubernetes assigned to the peer1- pod.
yacovm (Fri, 12 Jul 2019 09:37:09 GMT):
> and it is accessed externally through a node port at localhost:30020.
how can it be accessed externally via localhost? @aatkddny
aatkddny (Fri, 12 Jul 2019 12:27:20 GMT):
@yacovm This is a single node k8s install running locally on my mac.
The fabric touchpoints - peers and orderers are exposed as nodeports.
This part of the my application is running on the same machine and using the HFClient in the java sdk to create a channel dynamically - which appears to work - and then load and instantiate chaincode on the same - which is running into problems. You need to read the second entry to get to where my real issue lies here.
And anticipating your question - the sdk doesn't support using an ingress. There's a chunk of code with a TODO in it for same.
aatkddny (Fri, 12 Jul 2019 12:27:20 GMT):
@yacovm This is a single node k8s install running locally on my mac.
The fabric touchpoints - peers and orderers are exposed as nodeports.
This part of the my application is running on the same machine and using the HFClient in the java sdk to create a channel dynamically - which appears to work - and then load and instantiate chaincode on the same - which is running into problems. You need to read the second entry that it didn't embed to get to where my real issue lies here.
And anticipating your question - the sdk doesn't support using an ingress. There's a chunk of code with a TODO in it for same.
aatkddny (Fri, 12 Jul 2019 12:27:20 GMT):
@yacovm This is a single node k8s install running locally on my mac.
The fabric touchpoints - peers and orderers are exposed as nodeports.
This part of the my application is running on the same machine and using the HFClient in the java sdk to create a channel dynamically - which appears to work - and then load and instantiate chaincode on the same - which is running into problems. You need to read the second entry that it didn't embed to get to where my real issue lies here.
And anticipating your question - the sdk doesn't support using an ingress. There's a chunk of code with a TODO in it for same.
Edit: This one
https://chat.hyperledger.org/channel/fabric-sdk-java?msg=Gnr59mPKqk8z3jzsT
aatkddny (Fri, 12 Jul 2019 12:27:20 GMT):
@yacovm This is a single node k8s install running locally on my mac.
The fabric touchpoints - peers and orderers are exposed as nodeports.
This part of the my application is running on the same machine and using the HFClient in the java sdk to create a channel dynamically - which appears to work - and then load and instantiate chaincode on the same - which is running into problems. You need to read the second entry that it didn't embed to get to where my real issue lies here.
And anticipating your question - the sdk doesn't support using an ingress. There's a chunk of code with a TODO in it for same.
Edit: This one
https://chat.hyperledger.org/channel/fabric-sdk-java?msg=Gnr59mPKqk8z3jzsT
To help clarify - inside k8s the two peers for this org - peer0-org and peer1-org are at 10.1.6.240 and 10.1.6.241 respectively.
And since I spent a little time looking at it this morning, the problem here was that I switched to using core.yaml to define the peer and the chaincode listen address was commented out.
Setting it to localhost:7052 fixes this error and gets me to this one instead. This is the log from peer1- now at 10.1.6.243 with peer0- at 10.1.6.242
```
2019-07-12 12:41:51.595 UTC [gossip.channel] reportMembershipChanges -> INFO 04c Membership view has changed. peers went online: [[peer0-org:30020 10.1.6.242:7051]] , current view: [[peer0-org:30020 10.1.6.242:7051]]
2019-07-12 12:52:23.372 UTC [comm.grpc.server] 1 -> INFO 04d unary call completed grpc.service=discovery.Discovery grpc.method=Discover grpc.peer_address=192.168.65.3:56160 grpc.code=OK grpc.call_duration=9.8847ms
2019-07-12 12:52:32.909 UTC [comm.grpc.server] 1 -> INFO 04e unary call completed grpc.service=discovery.Discovery grpc.method=Discover grpc.peer_address=192.168.65.3:56160 grpc.code=OK grpc.call_duration=1.7358ms
2019-07-12 12:52:37.909 UTC [core.comm] ServerHandshake -> ERRO 04f TLS handshake failed with error read tcp 10.1.6.243:7051->192.168.65.3:56202: i/o timeout server=PeerServer remoteaddress=192.168.65.3:56202
2019-07-12 12:52:37.909 UTC [grpc] handleRawConn -> DEBU 050 grpc: Server.Serve failed to complete security handshake from "192.168.65.3:56202": read tcp 10.1.6.243:7051->192.168.65.3:56202: i/o timeout
2019-07-12 12:52:41.606 UTC [grpc] infof -> DEBU 051 transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2019-07-12 12:52:41.606 UTC [grpc] infof -> DEBU 052 transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2019-07-12 12:52:41.606 UTC [grpc] infof -> DEBU 053 transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2019-07-12 12:52:41.609 UTC [comm.grpc.server] 1 -> INFO 054 streaming call completed grpc.service=protos.Deliver grpc.method=Deliver grpc.peer_address=192.168.65.3:56182 error="context finished before block retrieved: context canceled" grpc.code=Unknown grpc.call_duration=13.8392412s
2019-07-12 12:52:41.612 UTC [comm.grpc.server] 1 -> INFO 055 streaming call completed grpc.service=protos.Deliver grpc.method=Deliver grpc.peer_address=192.168.65.3:56162 error="context finished before block retrieved: context canceled" grpc.code=Unknown grpc.call_duration=13.9075399s
2019-07-12 12:54:42.870 UTC [core.comm] ServerHandshake -> ERRO 056 TLS handshake failed with error read tcp 10.1.6.243:7051->192.168.65.3:56582: i/o timeout server=PeerServer remoteaddress=192.168.65.3:56582
2019-07-12 12:54:42.871 UTC [grpc] handleRawConn -> DEBU 057 grpc: Server.Serve failed to complete security handshake from "192.168.65.3:56582": read tcp 10.1.6.243:7051->192.168.65.3:56582: i/o timeout
```
The error in the SDK is coming from the X509 Cert - `java.io.IOException: No extension found with name NetscapeCertType`
It's in the peer eventing stuff, which means the machine keeps throwing threads as you try to look at it, but it all stems from it trying to send a discovery request.
aatkddny (Fri, 12 Jul 2019 12:27:20 GMT):
@yacovm This is a single node k8s install running locally on my mac.
The fabric touchpoints - peers and orderers are exposed as nodeports.
This part of the my application is running on the same machine and using the HFClient in the java sdk to create a channel dynamically - which appears to work - and then load and instantiate chaincode on the same - which is running into problems. You need to read the second entry that it didn't embed to get to where my real issue lies here.
And anticipating your question - the sdk doesn't support using an ingress. There's a chunk of code with a TODO in it for same.
Edit: This one
https://chat.hyperledger.org/channel/fabric-sdk-java?msg=Gnr59mPKqk8z3jzsT
To help clarify - inside k8s the two peers for this org - peer0-org and peer1-org were at 10.1.6.240 and 10.1.6.241 respectively.
And since I spent a little time looking at it this morning, the problem here was that I switched to using core.yaml to define the peer and the chaincode listen address was commented out.
Setting it to localhost:7052 fixes this error and gets me to this one instead. This is the log from peer1- now at 10.1.6.243 with peer0- at 10.1.6.242
```
2019-07-12 12:41:51.595 UTC [gossip.channel] reportMembershipChanges -> INFO 04c Membership view has changed. peers went online: [[peer0-org:30020 10.1.6.242:7051]] , current view: [[peer0-org:30020 10.1.6.242:7051]]
2019-07-12 12:52:23.372 UTC [comm.grpc.server] 1 -> INFO 04d unary call completed grpc.service=discovery.Discovery grpc.method=Discover grpc.peer_address=192.168.65.3:56160 grpc.code=OK grpc.call_duration=9.8847ms
2019-07-12 12:52:32.909 UTC [comm.grpc.server] 1 -> INFO 04e unary call completed grpc.service=discovery.Discovery grpc.method=Discover grpc.peer_address=192.168.65.3:56160 grpc.code=OK grpc.call_duration=1.7358ms
2019-07-12 12:52:37.909 UTC [core.comm] ServerHandshake -> ERRO 04f TLS handshake failed with error read tcp 10.1.6.243:7051->192.168.65.3:56202: i/o timeout server=PeerServer remoteaddress=192.168.65.3:56202
2019-07-12 12:52:37.909 UTC [grpc] handleRawConn -> DEBU 050 grpc: Server.Serve failed to complete security handshake from "192.168.65.3:56202": read tcp 10.1.6.243:7051->192.168.65.3:56202: i/o timeout
2019-07-12 12:52:41.606 UTC [grpc] infof -> DEBU 051 transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2019-07-12 12:52:41.606 UTC [grpc] infof -> DEBU 052 transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2019-07-12 12:52:41.606 UTC [grpc] infof -> DEBU 053 transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2019-07-12 12:52:41.609 UTC [comm.grpc.server] 1 -> INFO 054 streaming call completed grpc.service=protos.Deliver grpc.method=Deliver grpc.peer_address=192.168.65.3:56182 error="context finished before block retrieved: context canceled" grpc.code=Unknown grpc.call_duration=13.8392412s
2019-07-12 12:52:41.612 UTC [comm.grpc.server] 1 -> INFO 055 streaming call completed grpc.service=protos.Deliver grpc.method=Deliver grpc.peer_address=192.168.65.3:56162 error="context finished before block retrieved: context canceled" grpc.code=Unknown grpc.call_duration=13.9075399s
2019-07-12 12:54:42.870 UTC [core.comm] ServerHandshake -> ERRO 056 TLS handshake failed with error read tcp 10.1.6.243:7051->192.168.65.3:56582: i/o timeout server=PeerServer remoteaddress=192.168.65.3:56582
2019-07-12 12:54:42.871 UTC [grpc] handleRawConn -> DEBU 057 grpc: Server.Serve failed to complete security handshake from "192.168.65.3:56582": read tcp 10.1.6.243:7051->192.168.65.3:56582: i/o timeout
```
The error in the SDK is coming from the X509 Cert - `java.io.IOException: No extension found with name NetscapeCertType`
It's in the peer eventing stuff, which means the machine keeps throwing threads as you try to look at it, but it all stems from it trying to send a discovery request.
yacovm (Fri, 12 Jul 2019 13:29:16 GMT):
@aatkddny so I opened a JIRA to make peers publish custom endpoints for discovery to publish along with what it publishes now
yacovm (Fri, 12 Jul 2019 13:29:25 GMT):
so you can configure peers to use localhost and the ports
yacovm (Fri, 12 Jul 2019 13:29:28 GMT):
would that help you?
aatkddny (Fri, 12 Jul 2019 13:49:41 GMT):
I really need the java sdk to support an ingress tbh. I have a jira for that in already. FAB-15877.
This problem - now I have a few minutes to look at it - is coming from service discovery in peer eventing inside the java sdk.
I'll move my observations and question over there. It's out of scope for this.
My question here still stands though.
I know the java sdk is the read-headed stepchild of fabric, but has anyone managed to get it working talking to the latest hlf inside kubernetes.
By working I mean up to and including adding orgs dynamically, creating channels and installing and instantiating chaincode working.
Ours was fine until moving to raft - mostly because we glossed over the complexities of securing everything. Now it's been a litany of problems and a total time suck.
helgaw (Fri, 12 Jul 2019 19:55:35 GMT):
Has joined the channel.
yacovm (Fri, 12 Jul 2019 20:53:58 GMT):
@aatkddny you opened it for Fabric which is used for Fabric core, but it is a java SDK issue
yacovm (Fri, 12 Jul 2019 20:54:25 GMT):
I moved it to the Java SDK project for you
aatkddny (Fri, 12 Jul 2019 22:41:06 GMT):
I thought I put it under the java sdk. Oops.
harsh-98 (Sat, 13 Jul 2019 06:33:33 GMT):
t is giving error:
```Error: could not assemble transaction, err proposal response was not successful, error code 500, msg timeout expired while starting chaincode mycc:1.0 for transaction```
https://lists.hyperledger.org/g/fabric/topic/30469542
Any solution
ygnr (Mon, 15 Jul 2019 08:21:43 GMT):
Anyone deployed a network with nodes on two different kubernetes clusters and communicate over the internet? How does peer to peer communication work and where are the settings to give the IP address of nodes?
AndresMartinezMelgar.itcl (Wed, 17 Jul 2019 06:53:49 GMT):
Hello, does anyone know a tutorial to implement a network of multiple nodes with kubernetes?
iramiller (Wed, 17 Jul 2019 17:07:41 GMT):
@AndresMartinezMelgar.itcl multiple nodes meaning multiple Kubernetes clusters? That is going to be a complex thing to do at least until FAB-15877 is completed and/or support for ingress methods is in place.
pankajcheema (Thu, 18 Jul 2019 03:59:44 GMT):
@iramiller I have deployed it
AndresMartinezMelgar.itcl (Thu, 18 Jul 2019 06:06:49 GMT):
Hi iramiller
I do not mean multiple Kubernetes clusters, but rather that the network is distributed on several computers and that the information is shared through the network, not a shared folder.
The vast majority of examples I have seen are done with docker using shared volumes for the information (so the network is not distributed but is centralized)
iramiller (Thu, 18 Jul 2019 14:50:53 GMT):
I believe the helm charts that someone made will do away with the shared volume configuration... those are published in the central helm registry and are a good reference if you are just getting started building up a HLF install in kubernetes.
iramiller (Thu, 18 Jul 2019 14:52:48 GMT):
@pankajcheema -- we have a dozen different cloud instances for member orgs all working together each with 3 peers, etc. distributed across Azure, AWS, and GKE ... it is certainly possible to do but not something that should be considered straight forward.
sahilgoel (Fri, 19 Jul 2019 05:45:28 GMT):
https://stackoverflow.com/questions/57106103/kubeadm-setting-up-cluster-on-aws
sahilgoel (Fri, 19 Jul 2019 05:45:32 GMT):
Any comment on the same?
JorgeNavarro (Mon, 29 Jul 2019 08:49:01 GMT):
Hello, is it possible to set a specific hard disk to a specific peer in GCP? Because when i try to create volumes with their own hard disks in GCP, my peers take a random disk, not the disk i wanted...
sahilgoel (Tue, 30 Jul 2019 06:11:11 GMT):
https://stackoverflow.com/questions/57265280/hyperledger-fabric-unable-to-invoke-using-node-sdk
sahilgoel (Tue, 30 Jul 2019 06:11:28 GMT):
Please have a look at the above question
Khaled.MH (Tue, 30 Jul 2019 10:56:57 GMT):
Has joined the channel.
ibanfi (Tue, 30 Jul 2019 15:18:23 GMT):
Has joined the channel.
iramiller (Tue, 30 Jul 2019 20:02:02 GMT):
do mean some sort of actual specific physical disk or simply assign a consistent PVC/PV ?
JorgeNavarro (Thu, 01 Aug 2019 07:21:32 GMT):
i mean, i assign a disk to a pv, but if i have more than one pv with a disk assigned, my pvcs goes to a pv randomly, can we specify in which pv the pvc should go?
iramiller (Thu, 01 Aug 2019 16:41:10 GMT):
you could get specific with sizes ... but I am not sure why you care which pv goes to which PVC as they have to be bound first before they can be used and once bound they should stay linked...
aatkddny (Mon, 05 Aug 2019 14:23:19 GMT):
I need a little help to figure out where to stick a jira.
Has anyone had any luck with service discovery and k8s at all?
The reason I'm asking pertains to my post a few above this. If I run without SD I can install and instantiate some chain code (I'll get round to actually testing it later today).
If I run with SD it fails miserably with a bunch of this stuff
```
2019-07-12 12:52:23.372 UTC [comm.grpc.server] 1 -> INFO 04d unary call completed grpc.service=discovery.Discovery grpc.method=Discover grpc.peer_address=192.168.65.3:56160 grpc.code=OK grpc.call_duration=9.8847ms
2019-07-12 12:52:32.909 UTC [comm.grpc.server] 1 -> INFO 04e unary call completed grpc.service=discovery.Discovery grpc.method=Discover grpc.peer_address=192.168.65.3:56160 grpc.code=OK grpc.call_duration=1.7358ms
2019-07-12 12:52:37.909 UTC [core.comm] ServerHandshake -> ERRO 04f TLS handshake failed with error read tcp 10.1.6.243:7051->192.168.65.3:56202: i/o timeout server=PeerServer remoteaddress=192.168.65.3:56202
2019-07-12 12:52:37.909 UTC [grpc] handleRawConn -> DEBU 050 grpc: Server.Serve failed to complete security handshake from "192.168.65.3:56202": read tcp 10.1.6.243:7051->192.168.65.3:56202: i/o timeout
```
aatkddny (Mon, 05 Aug 2019 14:23:19 GMT):
I just got back to HLF. Had to take a detour onto a different project.
I need a little help to figure out where to stick a jira.
Has anyone had any luck with service discovery, raft ordering and k8s at all?
The reason I'm asking pertains to my post a few above this. I found that if I run without SD I can install and instantiate chaincode just the same as I could before I detoured into raft hell. (I'll get round to actually testing it later today).
If I run with SD it fails miserably with a bunch of this stuff
```
2019-07-12 12:52:23.372 UTC [comm.grpc.server] 1 -> INFO 04d unary call completed grpc.service=discovery.Discovery grpc.method=Discover grpc.peer_address=192.168.65.3:56160 grpc.code=OK grpc.call_duration=9.8847ms
2019-07-12 12:52:32.909 UTC [comm.grpc.server] 1 -> INFO 04e unary call completed grpc.service=discovery.Discovery grpc.method=Discover grpc.peer_address=192.168.65.3:56160 grpc.code=OK grpc.call_duration=1.7358ms
2019-07-12 12:52:37.909 UTC [core.comm] ServerHandshake -> ERRO 04f TLS handshake failed with error read tcp 10.1.6.243:7051->192.168.65.3:56202: i/o timeout server=PeerServer remoteaddress=192.168.65.3:56202
2019-07-12 12:52:37.909 UTC [grpc] handleRawConn -> DEBU 050 grpc: Server.Serve failed to complete security handshake from "192.168.65.3:56202": read tcp 10.1.6.243:7051->192.168.65.3:56202: i/o timeout
```
If I look in the peer logs it doesn't seem to like the IP address that it has - it's giving a bunch of stuff that looks like this `authentication handshake failed: x509: certificate is valid for 127.0.0.1, not 10.1.6.241`
So my question boils down to whether this has worked for anyone that doesn't use the java sdk, or if it is an actual hyperledger - vs that particular sdk - issue. It obviously needs to be fixed, I'm just unsure which particular group needs to know about it.
aatkddny (Tue, 06 Aug 2019 00:27:31 GMT):
NM - it happens with a kafka setup without tls too. Can't imagine that hasn't been tested, so it's most likely either the SDK or user error.
ihormudryy (Mon, 12 Aug 2019 13:14:26 GMT):
Hello! Two questions: 1. What would happen to instantiated chaincodes and the ledger itself if peer is restarted on different pod or being rescheduled? Did anyone test such behaviour in k8s. 2. Is there way do decouple peer's state DB to keep it separately as persistent data?
iramiller (Mon, 12 Aug 2019 16:00:07 GMT):
@ihormudryy you will want to setup your kubernetes peer deployments with all of their state on mounted volumes so that when the peer restarts / moves to another node the state information follows. If you are using the <=1.4.x series of HLF with peer managed chain code instances scheduled via docker on the host itself then you will likely find chaincode instance management within a kubernetes cluster to have some additional challenges especially if you are on small instances where resource limitations come into play. For our environments we patched that docker scheduling of chaincode containers out for one that used kubernetes managed deployments so I can not provide much advice on those pieces. There are others in this channel that are running the stock HLF though with success so it can be done.
aatkddny (Thu, 15 Aug 2019 00:18:54 GMT):
WRT the state database - you can run a couch database for state and map to a pvc. We run in the peer node as a secondary container.
I'll throw you a bone - the config looks like this for a single peer-db instance.
```
- name: blockcouchdbpeer1
image: hyperledger/fabric-couchdb
env:
- name: COUCHDB_USER
value: peer1
- name: COUCHDB_PASSWORD
value: password
volumeMounts:
- mountPath: /opt/couchdb/data
name: peer
subPath: couch/blockcouchdbpeer1
readinessProbe:
initialDelaySeconds: 10
timeoutSeconds: 10
httpGet:
path: /
port: 5984
scheme: HTTP
ports:
- containerPort: 5984
restartPolicy: Always
volumes:
- name: peer
persistentVolumeClaim:
claimName: bc-storage
```
aatkddny (Thu, 15 Aug 2019 00:18:54 GMT):
WRT the state database - you can run a couch database for state and map to a pvc. We run in the peer node as a secondary container.
I'll throw you a bone - the config looks like this for a single peer-db instance in my local test setup. Obviously it's a bit more secure from a user/password perspective elsewhere.
```
- name: blockcouchdbpeer1
image: hyperledger/fabric-couchdb
env:
- name: COUCHDB_USER
value: peer1
- name: COUCHDB_PASSWORD
value: password
volumeMounts:
- mountPath: /opt/couchdb/data
name: peer
subPath: couch/blockcouchdbpeer1
readinessProbe:
initialDelaySeconds: 10
timeoutSeconds: 10
httpGet:
path: /
port: 5984
scheme: HTTP
ports:
- containerPort: 5984
restartPolicy: Always
volumes:
- name: peer
persistentVolumeClaim:
claimName: bc-storage
```
aatkddny (Thu, 15 Aug 2019 00:22:27 GMT):
WRT 1: iirc - to get round the chaincode issue we map the dind directory to persistent (local on my machine or nfs in a cluster) storage. It seems to survive peer restarts without too much angst that way. I'll need to go back and see what we did for that - it was a while ago - if you can't figure it out.
ihorbilovus (Thu, 15 Aug 2019 14:07:42 GMT):
Has joined the channel.
ihorbilovus (Thu, 15 Aug 2019 14:07:44 GMT):
@aatkddny
do you use docker in docker for peers?
aatkddny (Thu, 15 Aug 2019 14:08:43 GMT):
yes
ihorbilovus (Thu, 15 Aug 2019 14:11:57 GMT):
Why do you use NFS for DinD storage. Can we use just persistent directory or this storage should be shared between all peers?
aatkddny (Thu, 15 Aug 2019 14:34:03 GMT):
Because I can't control which peer it gets mapped to. This allows me to not care if it restarts on another peer.
galaxystar (Tue, 20 Aug 2019 01:22:57 GMT):
Has joined the channel.
Chandoo (Tue, 20 Aug 2019 16:32:44 GMT):
I would like to know how you guyz are dealing with "." notation with the hostnames in kubernetes deployment
Chandoo (Tue, 20 Aug 2019 16:33:22 GMT):
How the dns alises being used in the deployment
iramiller (Tue, 20 Aug 2019 16:40:21 GMT):
@Chandoo ... We use `pod_name.namespace` i.e. `peer-2.memberorgname` in our environments ... we route all of our blockchain platform call on the inside of the Kubernetes network and do not expose these services directly on the internet which makes things much easier...
Chandoo (Tue, 20 Aug 2019 16:47:59 GMT):
@iramiller : thank you, I got it
Chandoo (Tue, 20 Aug 2019 16:50:36 GMT):
I am migrating from docker swarm to kubernetes. in swarm they created wired names that includes hostname.env.extraname.domainname, all the bootstrapping scripts are being auto generated with that wired name. I am dealing with it right now. I am recommending changes to similar to what you suggested.
Chandoo (Tue, 20 Aug 2019 16:51:50 GMT):
@iramiller : how you are routing the service layer that is comming from internet into the cluster reaching the orderer
iramiller (Tue, 20 Aug 2019 17:00:16 GMT):
@Chandoo we use a wireguard VPN connection on a host that is dual homed with the pod network/public internet in each of our member cloud environments and then point other member services (peers, ca, etc as well as orderers) using the `service.membername` format for services that have custom endpoints point to addresses on the wireguard vpn IP ... which then forwards this traffic directly into the destination member cluster to the kubernetes service reference that points to the pod/container instance ... this in effect means that any member can resolve any other member's services in any of the other clouds [GKE, EKS, AKS] and route directly to them all while using a fully encrypted single IP/port tunnel interface
iramiller (Tue, 20 Aug 2019 17:02:08 GMT):
This environment has worked very well for us especially with certain banking partners that have extremely tight network access control policies. We have well over a dozen different member org instances in different public clouds/availability zones all over that are built on this setup.
Chandoo (Tue, 20 Aug 2019 17:41:54 GMT):
@iramiller : thanks for the elaborate explanation.
Salaria_77 (Wed, 21 Aug 2019 07:14:55 GMT):
Has joined the channel.
Salaria_77 (Wed, 21 Aug 2019 07:18:36 GMT):
Hi all, i am not able to create replicas for the peer, i have running network it works fine , but when i increase replicas fir the peer it gives following error, looks like only one peer is able to establish connection with single peer.
panic: Error opening leveldb: resource temporarily unavailable
Salaria_77 (Wed, 21 Aug 2019 07:18:36 GMT):
Hi all, i am not able to create replicas for the peer, i have running network it works fine , but when i increase replicas the peer it gives following error, looks like only one peer is able to establish connection with single peer.
panic: Error opening leveldb: resource temporarily unavailable
Salaria_77 (Wed, 21 Aug 2019 07:18:36 GMT):
Hi all, i am not able to create replicas for the peer, i have running network it works fine , but when i increase replicas of the peer it gives following error, looks like only one peer is able to establish connection with level db.
panic: Error opening leveldb: resource temporarily unavailable
Salaria_77 (Wed, 21 Aug 2019 07:18:36 GMT):
Hi all, i am not able to create replicas for the peer, i have running network it works fine , but when i increase replicas of the peer it gives following error, looks like only one peer is able to establish connection with level db.
panic: Error opening leveldb: resource temporarily unavailable.
iramiller (Thu, 22 Aug 2019 16:35:00 GMT):
each peer should have its own state store @Salaria_77 ... are you trying to share a single instance between more than one peer?
Salaria_77 (Fri, 23 Aug 2019 04:39:15 GMT):
@iramiller No i am creating replicas of the single peer. I have a org with single peer0 and i am creating 3 replicas of single peer0, but couchdb is only addressing single replica at a time.
Salaria_77 (Fri, 23 Aug 2019 04:39:15 GMT):
@iramiller No i am creating replicas of the single peer. I have a org with single peer0 and i am creating 3 replicas of single peer0, but couchdb is only addressing single replica at a time. It looks like concurrency is not supported.
iramiller (Fri, 23 Aug 2019 14:41:41 GMT):
make sure that you are separating your peers and couchdb instances by namespace (one namespace for each org) and then verify your peer configuration uses `couchdb.orgnamespace` for the connection
Salaria_77 (Mon, 26 Aug 2019 04:41:57 GMT):
I am working with only one organization right now.
iramiller (Mon, 26 Aug 2019 14:33:47 GMT):
If you are only working with a single org then you would have a single namespace ... and each peer should be uniquely named, and have its own state store ... in my setup I use a stateful set with each instance having two containers--one peer and one couchdb. This approach ensure that the state store is local to the peer instance and provides some additional scheduling restrictions within kubernetes for how the instances are scheduled and disks are assigned.
iramiller (Mon, 26 Aug 2019 14:35:00 GMT):
With the above setup set to 3 replicas I have `peer-0.namespace`, `peer-1.namespace` and `peer-2.namespace` where the `namespace` is the name of my organization (by convention)
aatkddny (Mon, 26 Aug 2019 21:33:39 GMT):
You can always stick a couchdb instance inside each pod for the peers. That way it'll self-segregate.
iramiller (Mon, 26 Aug 2019 23:17:58 GMT):
@aatkddny - that is what I am referring to with the two container approach above. This setup means it is a localhost call to talk to couchdb which is a useful optimization as well.
aatkddny (Tue, 27 Aug 2019 12:03:22 GMT):
@iramiller that's what i figured, but the bit where you started talking about namespacing confused me to the point that I wrote my own response...
Salaria_77 (Tue, 27 Aug 2019 12:14:36 GMT):
@iramiller - It means i have to add new peers to the org with newly generated crypto material and those peers will be identified as diff entities inside the organisation. What i want is to use replicas feature in deployment or statefulset so that i can have single peer entity replicated inside the cluster.
Salaria_77 (Tue, 27 Aug 2019 12:24:07 GMT):
@iramiller - I think in my case one peer replica is locking the couch or level db, that's why only one replica work at one time reset enter CrashLoopBackOff .
iramiller (Tue, 27 Aug 2019 14:52:58 GMT):
@Salaria_77 if you are trying use a single peer registration and the replicas feature to load balance or provide fault tolerance I don't believe you are going to have any success... those questions should probably be posed to the #fabric-peer-endorser-committer channel.
aatkddny (Tue, 27 Aug 2019 18:39:46 GMT):
just aheads up in case anyone is using - or gets "upgraded" to a newer version of docker:dind as a sidecar.
be aware that docker set tls as a default after 18.09 so if you were using port 2375 without any env overrides it stops working unless you revert to an older version.
Salaria_77 (Wed, 28 Aug 2019 03:51:44 GMT):
@iramiller @aatkddny - Thanks for you time.
RuiPanNewbie (Thu, 29 Aug 2019 19:36:32 GMT):
Has joined the channel.
DennisM330 (Sun, 01 Sep 2019 15:07:15 GMT):
Can Fabric run on Redhat OpenShift?
yacovm (Sun, 01 Sep 2019 15:14:37 GMT):
OpenShift is a cloud platform for containers isn't it?
yacovm (Sun, 01 Sep 2019 15:14:59 GMT):
if you know how to run Fabric on openShift then it probably can
yacovm (Sun, 01 Sep 2019 15:15:06 GMT):
i don't think there is something that limits you
AnkurDaharwal (Tue, 03 Sep 2019 09:36:20 GMT):
Has joined the channel.
tbrunain (Wed, 04 Sep 2019 07:21:31 GMT):
Has joined the channel.
tbrunain (Fri, 06 Sep 2019 11:09:06 GMT):
Hello,
I’m currently trying to deploy a fabric network on kubernetes (which I’m learning while doing that so I might have missed something ^^) and I’m struggling with one part …
I don’t want to use cryptogen to pre-generate all the certificates and then mount them into the different pods but use fabric-ca to generate them .
So … how the hell am I supposed to do that ? Like ok, I can generate the certificates via the fabric-client enroll command on those CA but then how can I take the generated certificates and put them into the peer/orderer pods ?
Thanks :)
tbrunain (Fri, 06 Sep 2019 11:09:06 GMT):
Hello,
I’m currently trying to deploy a fabric network on kubernetes (which I’m learning while doing that so I might have missed some key concepts ^^) and I’m struggling with one part …
I don’t want to use cryptogen to pre-generate all the certificates and then mount them into the different pods but use fabric-ca to generate them .
So … how the hell am I supposed to do that ? Like ok, I can generate the certificates via the fabric-client enroll command on those CA but then how can I take the generated certificates and put them into the peer/orderer pods ?
Thanks :)
tbrunain (Fri, 06 Sep 2019 11:40:13 GMT):
Or am I wrong by thinking that I need to run the fabric-ca-client enroll directly onto the CA ? Definitely something I'm missing in there :x
mastersingh24 (Fri, 06 Sep 2019 12:55:54 GMT):
You can create a kubernetes secret from an MSP folder on your local system and then mount that as a volume on your peer and orderer nodes. See https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-files-from-a-pod for more info on using secrets as files.
You can also use an init container to run the fabric-ca-client to enroll the user and generate the secret there as well.
You can also choose to only store the private key as a secret and all of the public key material can be stored on a mounted persistent volume
mwagner (Fri, 06 Sep 2019 13:58:47 GMT):
@DennisM330 assuming you are running Fabric V1.x you will hit several issues around permissions and the docker in docker problem. These workarounds have worked for me.
mwagner (Fri, 06 Sep 2019 13:58:53 GMT):
Look to use Secrets and ConfigMaps to replace host mounts
Use NFS mounts where needed
oc adm policy add-scc-to-user hostmount-anyuid -z default
Replace docker-compose, docker calls with:
kubectl , oc, podman,Buildah, kompose
Convert docker-compose.yaml files with kompose
kompose convert --provider=openshift -f
Then edit and merge files
mbanerjee (Fri, 06 Sep 2019 17:37:05 GMT):
Has joined the channel.
mbanerjee (Fri, 06 Sep 2019 17:37:51 GMT):
We have been struggling to get Fabric working on Kubernetes. Can some one please point to tutorials or best practices? TIA
mwagner (Fri, 06 Sep 2019 19:10:47 GMT):
did you try this ? https://github.com/IBM/blockchain-network-on-kubernetes
mastersingh24 (Sun, 08 Sep 2019 10:49:33 GMT):
What are your issues?
Dhiraj1990 (Mon, 09 Sep 2019 04:06:53 GMT):
HI !
What's issue you're facing ?
dsessions (Mon, 09 Sep 2019 17:53:14 GMT):
Has joined the channel.
dsessions (Mon, 09 Sep 2019 17:53:15 GMT):
HLF on kubernetes using more native methods of embedding MSP crypto material into secrets and config maps is a bit more of a dark art for the GO SDK clients. Core HLF components are fairly easy to get going with the hlf helm charts.
dsessions (Mon, 09 Sep 2019 17:53:15 GMT):
@mbanerjee HLF on kubernetes using more native methods of embedding MSP crypto material into secrets and config maps is a bit more of a dark art for the GO SDK clients. Core HLF components are fairly easy to get going with the hlf helm charts.
iramiller (Mon, 09 Sep 2019 17:57:25 GMT):
A useful technique I use for configmap/secret work in Kubernetes peers has been to use a custom startup script for the peer containers that performs some additional setup prior to invoking the peer command. This script itself is stored in a config map mounted into the container to avoid the need for custom packaging.
dsessions (Mon, 09 Sep 2019 17:58:50 GMT):
in my humble opinion, the hlf helm charts are 100000 x better than the IBM kubernetes examples
iramiller (Mon, 09 Sep 2019 17:59:10 GMT):
Depending on your requirements you can use this type of script to directly load kubernetes secrets/configmaps into your container (using `curl`) at startup versus the typical pod/container specification approach...
soumyanayak (Wed, 11 Sep 2019 14:52:09 GMT):
Has joined the channel.
abel23 (Thu, 12 Sep 2019 04:41:46 GMT):
Has joined the channel.
mbanerjee (Thu, 12 Sep 2019 17:36:56 GMT):
Ledger data that is stored in the peer node, how does that get backed up? Do we need to have a separate persistent disk for ledger data? Any suggestions. thanks.
dsessions (Thu, 12 Sep 2019 20:54:33 GMT):
Ledger data stored on the peer node? Are you talking about data stored in the couch db instance?
dsessions (Thu, 12 Sep 2019 20:54:50 GMT):
Ledger data stored on the peer node? Are you talking about data stored in the couch db instance?
dsessions (Thu, 12 Sep 2019 20:57:36 GMT):
If you are . . just run another peer to duplicate the data.
mbanerjee (Fri, 13 Sep 2019 01:52:51 GMT):
No the actual ledger
mbanerjee (Fri, 13 Sep 2019 01:52:55 GMT):
not the state database
mbanerjee (Fri, 13 Sep 2019 01:53:16 GMT):
What if node that the peer is running on dies?
mbanerjee (Fri, 13 Sep 2019 01:53:31 GMT):
The ledger data is being written in the peer node will also be lost
dsessions (Fri, 13 Sep 2019 01:59:27 GMT):
https://hyperledger-fabric.readthedocs.io/en/release-1.4/txflow.html
dsessions (Fri, 13 Sep 2019 01:59:38 GMT):
Checkout sections 4, 5, and 6
soumyanayak (Fri, 13 Sep 2019 05:38:05 GMT):
Ledger data can be backed up as either you make one more peer up as suggested by @dsessions .
or else if you want to take the back up of the peerledger folder completely in some remote servers you can check the below link
https://blockchain-fabric.blogspot.com/2019/02/hyperledger-fabric-ledger-backup-and.html
ahmad-raza (Fri, 13 Sep 2019 12:57:16 GMT):
Has joined the channel.
ahmad-raza (Fri, 13 Sep 2019 12:57:40 GMT):
Hello ALl, Anyone know about this error
"instantiate proposal resulted in an error :: Error: error starting container: error starting container: cannot connect to Docker endpoint"
I am using ibm-kubernetes-cluster. Everything is working untill instantiating the chaincode
ahmad-raza (Fri, 13 Sep 2019 12:57:58 GMT):
Screenshot from 2019-09-13 17-27-32.png
ahmad-raza (Fri, 13 Sep 2019 12:58:11 GMT):
this image is error logs from peer logs
ahmad-raza (Fri, 13 Sep 2019 12:58:25 GMT):
Failed to instantiate the chaincode. cause:instantiate proposal resulted in an error :: Error: error starting container: error starting container: cannot connect to Docker endpoint"
ahmad-raza (Fri, 13 Sep 2019 13:00:58 GMT):
certificates, channel creation , joining channel, installing chaincode on channels are done but error occured during instantiating chaincode. Does all other things not required "DOcker"?
ahmad-raza (Fri, 13 Sep 2019 15:05:21 GMT):
???
ahmad-raza (Fri, 13 Sep 2019 20:28:14 GMT):
Anyone can help?
ahmad-raza (Sun, 15 Sep 2019 19:07:57 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=dmLjftFEFmxSPb5Af) Any one can tell me how to instantiate chaincode in ibm kubernetes cluster. In local setup i did it successfully
ahmad-raza (Mon, 16 Sep 2019 07:35:47 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=vbjuH7gBNRNgvkdG4) ??
MohammedR (Mon, 16 Sep 2019 07:41:40 GMT):
its probably chain code container is not able to connect to peer container try adding GODEBUG: "netdns=go" to env
MohammedR (Mon, 16 Sep 2019 07:41:40 GMT):
its probably becau chain code container is not able to connect to peer container try adding GODEBUG: "netdns=go" to env
MohammedR (Mon, 16 Sep 2019 07:41:40 GMT):
its probably because chain code container is not able to connect to peer container try adding GODEBUG: "netdns=go" to env
MohammedR (Mon, 16 Sep 2019 07:41:40 GMT):
its probably because chain code container is not able to connect to peer container try adding GODEBUG: "netdns=go" to env of peer
ahmad-raza (Mon, 16 Sep 2019 07:56:42 GMT):
@MohammedR it this variable is for debuging purpose?
MohammedR (Mon, 16 Sep 2019 07:58:06 GMT):
yes
ahmad-raza (Mon, 16 Sep 2019 08:01:32 GMT):
Screenshot from 2019-09-16 13-01-14.png
ahmad-raza (Mon, 16 Sep 2019 08:01:43 GMT):
its generating error
MohammedR (Mon, 16 Sep 2019 08:02:14 GMT):
what is the error
ahmad-raza (Mon, 16 Sep 2019 08:02:48 GMT):
Screenshot from 2019-09-16 13-02-38.png
MohammedR (Mon, 16 Sep 2019 08:03:41 GMT):
name=GODEBUG
ahmad-raza (Mon, 16 Sep 2019 08:04:07 GMT):
i am using ibm kubernetes cluster. I have read somewhere that kubernetes does not use docker . we have to do something like docker deamon
ahmad-raza (Mon, 16 Sep 2019 08:04:40 GMT):
i have successfully setup network on my local machine with kubernetes single worker node but on ibm it gives error
MohammedR (Mon, 16 Sep 2019 08:05:29 GMT):
I didn't understand
ahmad-raza (Mon, 16 Sep 2019 08:06:24 GMT):
do i need to run any other container other than peers, orderers, ca's on ibm kubernetes
ahmad-raza (Mon, 16 Sep 2019 08:06:27 GMT):
?
MohammedR (Mon, 16 Sep 2019 08:06:35 GMT):
no
ahmad-raza (Mon, 16 Sep 2019 08:07:26 GMT):
i am following that link https://github.com/IBM/blockchain-network-on-kubernetes
ahmad-raza (Mon, 16 Sep 2019 08:08:00 GMT):
according to it we on kubernetess > 1.11 we have to specify docker-dind in it
ahmad-raza (Mon, 16 Sep 2019 08:17:20 GMT):
Screenshot from 2019-09-16 13-16-42.png
ahmad-raza (Mon, 16 Sep 2019 08:45:27 GMT):
@MohammedR any help?
MohammedR (Mon, 16 Sep 2019 09:41:16 GMT):
its not able to connect to your docker container try to ping docker container check if its reachable
ahmad-raza (Mon, 16 Sep 2019 09:42:02 GMT):
how? on ibm kubernetes cluster?
mastersingh24 (Mon, 16 Sep 2019 12:05:51 GMT):
Kubernetes clusters no longer run Docker as the container runtime so you can no longer try to mount the Doker socket on your peer pod
ahmad-raza (Mon, 16 Sep 2019 12:06:47 GMT):
@mastersingh24 thanks for answer any workaround?
mastersingh24 (Mon, 16 Sep 2019 12:06:48 GMT):
You will need to have a Docker daemon running somewhere ... either on a node outside your Kube cluster or by running Docker in a container (Docker in Docker)
ahmad-raza (Mon, 16 Sep 2019 12:08:00 GMT):
- name: dockervolume
persistentVolumeClaim:
claimName: docker-pvc
containers:
- name: docker
securityContext:
privileged: true
image: "docker:stable-dind"
ports:
- containerPort: 2375
volumeMounts:
- mountPath: /var/lib/docker
name: dockervolume
mastersingh24 (Mon, 16 Sep 2019 12:09:12 GMT):
and of course you need to configure the peer to be able to communicate with the DinD container / pod
ahmad-raza (Mon, 16 Sep 2019 12:09:39 GMT):
i am running this container in same pod where peer and couchdb container is created and provide
core_vm_endpoint=localhost:2375 but still cannot connect to end point
mastersingh24 (Mon, 16 Sep 2019 12:10:05 GMT):
localhost will not work
ahmad-raza (Mon, 16 Sep 2019 12:10:20 GMT):
clusterIP of pod?
ahmad-raza (Mon, 16 Sep 2019 12:10:38 GMT):
or nodeIP:nodeport?
ahmad-raza (Mon, 16 Sep 2019 12:13:44 GMT):
And am i have to make separate pod or container for separate namespaces?
mastersingh24 (Mon, 16 Sep 2019 12:14:57 GMT):
hmm ... actually I think localhost should work ... let me check
mastersingh24 (Mon, 16 Sep 2019 12:14:57 GMT):
hmm ... actually I think localhost should work since it's in the same pod as the peer
ahmad-raza (Mon, 16 Sep 2019 12:33:00 GMT):
but it does not work :disappointed:
mastersingh24 (Mon, 16 Sep 2019 12:40:20 GMT):
can you post your peer spec?
ahmad-raza (Mon, 16 Sep 2019 12:41:51 GMT):
ahmad-raza - Mon Sep 16 2019 17:41:46 GMT+0500 (Pakistan Standard Time).txt
mastersingh24 (Mon, 16 Sep 2019 13:00:36 GMT):
are you sure the docker container is actually coming up properly? You might want to remove all of the mounts / hostPath stuff from the docker spec
ahmad-raza (Mon, 16 Sep 2019 13:03:38 GMT):
Yes container it coming up. from container logs it shows it is listening
ahmad-raza (Mon, 16 Sep 2019 13:03:55 GMT):
logs-from-docker-in-docker-dind-6d74f44c75-cl2sq.txt
ahmad-raza (Mon, 16 Sep 2019 13:07:28 GMT):
from logs it shows some iocontainer.d files error but at ends it says
Api listen on [::]:2376
Api listen on /var/run/docker.sock
mastersingh24 (Mon, 16 Sep 2019 13:08:05 GMT):
oh ... note that is it 2376 and not 2375 ... looks like it is running using TLS
ahmad-raza (Mon, 16 Sep 2019 13:08:56 GMT):
i have tried 2376 but it gives tls handshake error
ahmad-raza (Mon, 16 Sep 2019 13:09:28 GMT):
But certs are fine all other operations are done through these certs
mastersingh24 (Mon, 16 Sep 2019 13:09:57 GMT):
right ... that is going to be trickier as you will need to add settings in the peer config for Docker TLS
ahmad-raza (Mon, 16 Sep 2019 13:10:56 GMT):
# - name: CORE_PEER_TLS_CLIENTROOTCAS_FILES
# value: /etc/hyperledger/fabric/tls/ca.crt
# - name: CORE_PEER_TLS_CLIENTCERT_FILE
# value: /etc/hyperledger/fabric/tls/server.crt
# - name: CORE_PEER_TLS_CLIENTKEY_FILE
# value: /etc/hyperledger/fabric/tls/server.key
ahmad-raza (Mon, 16 Sep 2019 13:11:28 GMT):
i added these lines are these are required?
mastersingh24 (Mon, 16 Sep 2019 13:11:46 GMT):
no ...
ahmad-raza (Mon, 16 Sep 2019 13:12:28 GMT):
?
mastersingh24 (Mon, 16 Sep 2019 13:14:40 GMT):
CORE_VM_DOCKER_TLS_ENABLED
CORE_VM_DOCKER_TLS_CA_FILE
and possibly
CORE_VM_DOCKER_TLS_CERT_FILE
CORE_VM_DOCKER_TLS_KEY_FILE
I'd actually advise trying without TLS first ...
ahmad-raza (Mon, 16 Sep 2019 13:16:18 GMT):
thanks i give them try . and i have to use same certs that are already in use right?
mastersingh24 (Mon, 16 Sep 2019 13:18:24 GMT):
So the dind image will automatically generate the certs for you (and by default it will only expose a TLS endpoint) ...
Looks like you can set `DOCKER_TLS_CERTDIR` for the dind container to use a shared directory for the generated certs
mastersingh24 (Mon, 16 Sep 2019 13:18:40 GMT):
The Dockerhub page has a good description of this
ahmad-raza (Mon, 16 Sep 2019 13:20:25 GMT):
ok i'll have look on it secondly if i am going without tls will i have to start things from scratch or simply disable it and instantiate chaincode
ahmad-raza (Mon, 16 Sep 2019 14:26:36 GMT):
Screenshot from 2019-09-16 19-25-30.png
ahmad-raza (Mon, 16 Sep 2019 14:26:53 GMT):
'CORE_TLS_CLIENT_KEY_PATH'
ahmad-raza (Mon, 16 Sep 2019 14:26:58 GMT):
@mastersingh24
mastersingh24 (Mon, 16 Sep 2019 14:32:28 GMT):
you only need the those if using you enable clientAuth for TLS ... it's disabled by default
ahmad-raza (Mon, 16 Sep 2019 14:34:17 GMT):
these are using for container creation . I have set environment variables you suggested but still x509: certificate signed by unknown authority
AndresMartinezMelgar.itcl (Mon, 16 Sep 2019 15:54:35 GMT):
hi,are anyone trying to make up your hyperledger fabric network in a own cluster( not using gpc, aws...)
ahmad-raza (Tue, 17 Sep 2019 09:13:29 GMT):
Any one can help
cause:upgrade proposal resulted in an error :: Error: chaincode registration failed: container exited with 254
Newer Chanicode version is installed on peers. But error while upgrading chaincode
Fabric network is setup on IBM kubernetes Cluster
dsessions (Tue, 17 Sep 2019 17:23:42 GMT):
have you compared the chaincode versions?
adityanalge (Tue, 17 Sep 2019 19:45:47 GMT):
Has joined the channel.
adityanalge (Tue, 17 Sep 2019 19:45:48 GMT):
How would you mount the genesis.block onto an orderer pod? Since the gensis.block filetype is 'data'
aatkddny (Wed, 18 Sep 2019 13:31:47 GMT):
Mount a share. Stick the block into that. If you are using core.yaml it's here - /share in this case is my share.
```
# Genesis file: The file containing the genesis block to use when
# initializing the orderer system channel and GenesisMethod is set to
# "file". Ignored if GenesisMethod is set to "provisional".
GenesisFile: /share/generated/channel-artifacts/genesis.block
```
nleut (Wed, 18 Sep 2019 15:00:15 GMT):
Has joined the channel.
mbanerjee (Wed, 18 Sep 2019 19:08:15 GMT):
does any one have a deployment for orderer with type raft? Do you use multiple containers in the same pod?
ahmad-raza (Thu, 19 Sep 2019 10:21:23 GMT):
@mbanerjee I am using multiple pods for that with one container each.
Bentipe (Thu, 19 Sep 2019 14:41:26 GMT):
Has joined the channel.
mbanerjee (Thu, 19 Sep 2019 15:38:41 GMT):
[36m2019-09-19 04:28:55.414 UTC [orderer.consensus.etcdraft] logSendFailure -> DEBU 3c1[0m Failed to send StepRequest to 4, because: connection to 4(orderer-service.default.svc.cluster.local:7053) is in state CONNECTING channel=network-sys-channel node=1=====
mbanerjee (Thu, 19 Sep 2019 15:38:57 GMT):
Seeing this issue when we start orderer with raft
ahmad-raza (Fri, 20 Sep 2019 05:52:47 GMT):
there may be issues with ports and container internal endpoints
iramiller (Fri, 20 Sep 2019 16:58:29 GMT):
@mbanerjee I have a statefulset deployment of orderers using raft running...
mbanerjee (Fri, 20 Sep 2019 18:25:15 GMT):
Getting this error when instantiating chaincode in Kubernetes - Failed to instantiate the chaincode. Reason: Instantiate chaincode proposal resulted in an error:: Error: error starting container: error starting container: Post http://docker:2375/containers/create?name=dev-peer0-org1-example-com-testcc5-v1: dial tcp: lookup docker on 10.4.0.10:53: no such host"
mbanerjee (Fri, 20 Sep 2019 18:25:15 GMT):
Getting this error when instantiating chaincode in Kubernetes - Failed to instantiate the chaincode. Reason: Instantiate chaincode proposal resulted in an error:: Error: error starting container: error starting container: Post http://docker:2375/containers/create?name=dev-peer0-org1-example-com-testcc5-v1: dial tcp: lookup docker on 10.4.0.10:53: no such host" @iramiller
ahmad-raza (Fri, 20 Sep 2019 21:36:16 GMT):
do you have dind?
ahmad-raza (Fri, 20 Sep 2019 21:36:37 GMT):
Docker in DOcker
mbanerjee (Fri, 20 Sep 2019 21:44:34 GMT):
Yes, we have dind
ahmad-raza (Mon, 23 Sep 2019 05:59:24 GMT):
which version of dind ? if you have latest , latest is tls enabled that listen on 2376 i think?
mbanerjee (Mon, 23 Sep 2019 21:14:17 GMT):
What all data from a network needs to be backed up? Certificates, peer ledger, state db? Do we need to back up any data from the orderers?
raj_shekhar (Tue, 24 Sep 2019 09:33:38 GMT):
Hi,
I have one query specific to Kubernetes, if the default service account of the kubernetes cluster got unbounded and if I restore it using below command as given in this link https://cloud.google.com/kubernetes-engine/docs/troubleshooting#gke_service_account_delete
will it delete the existing kubernetes cluster and data will be lost or it will just restore the service account and old cluster will be fine.??? I am using GCP.
@iramiller
raj_shekhar (Tue, 24 Sep 2019 11:36:27 GMT):
Sorted it ....... created a new IAM role binding for the kubernetes APIs... working fine
adityanalge (Tue, 24 Sep 2019 22:46:54 GMT):
What is the best way to configure the ORDERER_URL environment variable in the peer? I am currently using the Orderer POD IP. Wrapping the orderer in clusterIP service with protocol type TCP gives an error. I am not sure if grpcs works with service-name.namespace.svc.cluster.local IP
iramiller (Tue, 24 Sep 2019 22:48:40 GMT):
@adityanalge we use Kubernetes service names and DNS everywhere ... no pod ip addresses
adityanalge (Tue, 24 Sep 2019 22:51:49 GMT):
So you are not seeing any issues with GRPCS?
iramiller (Tue, 24 Sep 2019 22:52:53 GMT):
@adityanalge to be clear we are _not_ exposing those endpoints outside of the cluster ... but internal routing within GKE, AKS, EKS clusters as well as our Wireguard private network between clusters are all working just fine for GRPC ...
iramiller (Tue, 24 Sep 2019 22:53:11 GMT):
we use TLS and Client certificates as well
adityanalge (Tue, 24 Sep 2019 22:53:15 GMT):
okay, good to know. Thanks!
FernandaSartori (Wed, 25 Sep 2019 13:47:23 GMT):
Has joined the channel.
adityanalge (Wed, 25 Sep 2019 18:11:17 GMT):
How are you dealing with the no such host error? Say orderer0 is in a different pod and orderer1 is in a different pod, they will communicate through TLS. Their host and port names are configured in the configtx.yaml. However, this gives me a no such host error as the ip address for orderer1 is not known to orderer0 and vice versa. As the pods are ephemeral, there is no way to know in advance the IP with which orderer0 and orderer1 pods will be created
adityanalge (Wed, 25 Sep 2019 18:11:17 GMT):
How are you dealing with the no such host error? Say for RAFT, orderer0 is in a different pod and orderer1 is in a different pod, they will communicate through TLS. Their host and port names are configured in the configtx.yaml. However, this gives me a no such host error as the ip address for orderer1 is not known to orderer0 and vice versa. As the pods are ephemeral, there is no way to know in advance the IP with which orderer0 and orderer1 pods will be create
adityanalge (Wed, 25 Sep 2019 18:11:17 GMT):
How are you dealing with the no such host error? Say for RAFT, orderer0 is in a different pod and orderer1 is in a different pod, they will communicate through TLS. Their host and port names are configured in the configtx.yaml. However, this gives me a no such host error as the ip address for orderer1 is not known to orderer0 and vice versa. As the pods are ephemeral, there is no way to know in advance the IP with which orderer0 and orderer1 pods will be create @iramiller
iramiller (Wed, 25 Sep 2019 18:54:31 GMT):
@adityanalge no such host error indicates you have a DNS issue... do you have a kubernetes service in place for each orderer? if you have dns utils installed in a pod in the same namespace try doing a DNS lookup for your orderer instances ... they must resolve correctly for this to work
iramiller (Wed, 25 Sep 2019 19:01:14 GMT):
@adityanalge if it makes you feel better I have both raft and Kafka based ordering systems working in my environments so it is certainly possible...
adityanalge (Wed, 25 Sep 2019 19:03:16 GMT):
Please indulge me for a moment here. I think the solution will be, to use the service name and port in configtx.yaml as ```Addresses:
- blockchain-orderer:
iramiller (Wed, 25 Sep 2019 19:04:39 GMT):
```
Addresses:
- orderer-0.consensus:7050
- orderer-1.consensus:7050
- orderer-2.consensus:7050
- orderer-3.consensus:7050
- orderer-4.consensus:7050```
iramiller (Wed, 25 Sep 2019 19:05:04 GMT):
indeed that is how I have configured the orderer network in my channel configs.
iramiller (Wed, 25 Sep 2019 19:06:11 GMT):
and for services ... here is an example
iramiller (Wed, 25 Sep 2019 19:06:16 GMT):
```
apiVersion: v1
kind: Service
metadata:
annotations:
cloud.google.com/load-balancer-type: Internal
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
name: orderer-0
namespace: consensus
labels:
app: orderer-consensus
release: orderer-consensus
spec:
type: LoadBalancer
loadBalancerIP: x.x.x.x
ports:
- port: 7050
protocol: TCP
name: orderer
- port: 8443
protocol: TCP
name: operations
selector:
statefulset.kubernetes.io/pod-name: orderer-0
app: orderer-consensus
externalTrafficPolicy: Cluster```
iramiller (Wed, 25 Sep 2019 19:06:45 GMT):
(from a GKE instance)
adityanalge (Wed, 25 Sep 2019 19:16:35 GMT):
Thanks for the help. I was doing orderer-service.namespace.svc.cluster.local in configtx.yaml which was giving errors. I will try this approach
iramiller (Wed, 25 Sep 2019 19:17:38 GMT):
`.svc.cluster.local` is part of the standard dns suffix search order for kubernetes hosts
adityanalge (Wed, 25 Sep 2019 20:27:35 GMT):
Could you share a sample of your configtx.yaml file?
iramiller (Wed, 25 Sep 2019 20:28:10 GMT):
Which piece are you interested in? @adityanalge
adityanalge (Wed, 25 Sep 2019 20:28:26 GMT):
The Etcd Raft consenters and addresses
adityanalge (Wed, 25 Sep 2019 20:28:42 GMT):
``` EtcdRaft:
Consenters:
&a6
- Host: orderer0
Port: 7050```
adityanalge (Wed, 25 Sep 2019 20:29:19 GMT):
``` Addresses:
- orderer-service.default:port1
- orderer-service.default:port2
- orderer-service.default:port3
- orderer-service.default:port4
- orderer-service.default:port5```
adityanalge (Wed, 25 Sep 2019 20:30:42 GMT):
Using orderer-service.default in address, gives me a tls error saying the tls certificate is valid for orderer0 and not for orderer-service.
iramiller (Wed, 25 Sep 2019 20:32:21 GMT):
that is because as a typical kubernetes service the endpoint you are using points to any one of the set
iramiller (Wed, 25 Sep 2019 20:32:47 GMT):
but what you really want is a service that points ONLY to a single orderer instance with the same name and TLS
iramiller (Wed, 25 Sep 2019 20:33:03 GMT):
per my example above `orderer-0.consensus:7050`
adityanalge (Wed, 25 Sep 2019 20:33:11 GMT):
so service name has to match orderer name?
iramiller (Wed, 25 Sep 2019 20:33:22 GMT):
there is a service with the name `orderer-0`
iramiller (Wed, 25 Sep 2019 20:33:38 GMT):
there is a service with for each orderer ... so 5 of them
iramiller (Wed, 25 Sep 2019 20:33:53 GMT):
and the service selector is
```
selector:
statefulset.kubernetes.io/pod-name: orderer-0
app: orderer-consensus
externalTrafficPolicy: Cluster
```
iramiller (Wed, 25 Sep 2019 20:34:13 GMT):
for the `orderer-0` svc so it points to the correct pod
iramiller (Wed, 25 Sep 2019 20:35:00 GMT):
Which supports this config from a running channel
```
"ConsensusType": {
"mod_policy": "Admins",
"value": {
"metadata": {
"consenters": [
{
"client_tls_cert": "LS0tLS1CRUdJTiBDRVJUSUZJ...
"host": "orderer-0.consensus",
"port": 7050,
"server_tls_cert": "LS0tLS1CRUdJTiBDRVJUSUZJ...
},
```
adityanalge (Wed, 25 Sep 2019 20:35:25 GMT):
I see you are adding the namespace along with orderer0.
iramiller (Wed, 25 Sep 2019 20:35:39 GMT):
yes... that is my environment configuration
adityanalge (Wed, 25 Sep 2019 20:35:41 GMT):
Can I skip it if everything is under default name space?
iramiller (Wed, 25 Sep 2019 20:35:44 GMT):
yes
iramiller (Wed, 25 Sep 2019 20:36:02 GMT):
I recommend using a namespace matching your organization name for other reasons
iramiller (Wed, 25 Sep 2019 20:36:18 GMT):
but default is fine if you have a single org in your environment(s)
adityanalge (Wed, 25 Sep 2019 23:45:47 GMT):
I made a lot of mistakes, but finally got it to work
adityanalge (Wed, 25 Sep 2019 23:45:49 GMT):
Thanks
heenas06 (Thu, 26 Sep 2019 07:01:49 GMT):
Has joined the channel.
heenas06 (Thu, 26 Sep 2019 07:09:52 GMT):
Hi guys I am beginner in Kubernetes,can you guide me how to deploy hyperledger fabric on Kubernetes or anyone have a sample examples for this ....??
jona-sc (Thu, 26 Sep 2019 10:29:11 GMT):
Has joined the channel.
iramiller (Thu, 26 Sep 2019 14:39:21 GMT):
@heenas06 -- kubernetes is an extremely deep technical product on its own... hopefully you are not trying to learn how Hyperledger Fabric works at the same time. At any rate a good place to start researching how to run Hyperledger inside Kubernetes is to review the helm charts. https://github.com/helm/charts/tree/ce6f5f931b32d6b5d92080df9a59953dd10cdd4a/stable/hlf-peer
iramiller (Thu, 26 Sep 2019 14:40:12 GMT):
@heenas06 I feel like I should also mentioned that Hyperledger Fabric in the 1.x series is not an easy fit for the Kubernetes (or other cloud native) environments.
dsessions (Thu, 26 Sep 2019 16:00:50 GMT):
There are a ton of gotchas with Kubernetes . . especially once you try and start using fabric-sdk-go
iramiller (Thu, 26 Sep 2019 23:06:04 GMT):
All of the Kubernetes issues that Fabric has are surmountable if you are willing and able to put in enough effort. That said it might require a fork of the code base if you are working with 1.x... I hold out hope that Fabric 2.0 will make things much easier.
heenas06 (Fri, 27 Sep 2019 04:36:39 GMT):
Thanks......for the help ...
adityanalge (Fri, 27 Sep 2019 17:12:34 GMT):
I agree with what @iramiller has said. Almost every Fabric requirement can be satisfied by Kubernetes. I think what's lacking is a proper end to end tutorial/video designed specifically for beginners in both technologies.
adityanalge (Fri, 27 Sep 2019 17:15:08 GMT):
What is the best way to mount certificates onto an orderer/peer? It is my understanding that the certificates have to be present on the orderer/peer before the pod starts?
iramiller (Fri, 27 Sep 2019 19:02:00 GMT):
@adityanalge we use secrets/configmaps to bring the certificates into the pod. This type of approach positions the system well for integration with tools such as Hashicorp Vault that protect sensitive configuration in larger enterprise cloud environments
dsessions (Fri, 27 Sep 2019 21:57:07 GMT):
Try using the helm charts . . that's the best way to start out imho
ahmad-raza (Mon, 30 Sep 2019 07:28:13 GMT):
Hello all, I have two ibm kubernetes cluster , On one cluster network is running perfectly which includes 2 orgs with one peer each and 5 orderers (raft based)
I want to add 3rd org to network and channel. Requirement is this org will run in separate cluster(2nd) . Org is set on 2nd cluster .
Using this https://hyperledger-fabric.readthedocs.io/en/release-1.4/channel_update_tutorial.html new org is added in channel
Org3 joined the channel successfully. Chaincode is installed on all peers . But when call for upgrade chaincode(Instantiate) Error is returned "Could not found chaincode with name mycc"
I have confirmed that chaincode is installed on all nodes ..
In the logs of org3 peer(that is on another cluster) there are following errors
Err :connection error: desc = "transport: Error while dialing dial tcp: lookup orderer5.example on 172.21.0.10:53: no such host". Reconnecting...
Failed obtaining connection: could not connect to any of the endpoints: [{orderer.example:7050 [OrdererMSP]} {orderer4.example:7050 [OrdererMSP]} {orderer3.example:7050 [OrdererMSP]} {orderer2.example:7050 [OrdererMSP]} {orderer5.example:7050 [OrdererMSP]}]
[36m2019-09-30 06:46:16.785 UTC [grpc] func1 -> DEBU 761[0m Failed to dial orderer5.example:7050: context canceled; please retry.
ahmad-raza (Mon, 30 Sep 2019 07:28:13 GMT):
Hello all, I have two ibm kubernetes cluster , On one cluster network is running perfectly which includes 2 orgs with one peer each and 5 orderers (raft based)
I want to add 3rd org to network and channel. Requirement is this org will run in separate cluster(2nd) . Org is set on 2nd cluster .
Using this https://hyperledger-fabric.readthedocs.io/en/release-1.4/channel_update_tutorial.html new org is added in channel
Org3 joined the channel successfully. Chaincode is installed on all peers . But when call for upgrade chaincode(Instantiate) Error is returned "Could not found chaincode with name mycc"
I have confirmed that chaincode is installed on all nodes ..
In the logs of org3 peer(that is on another cluster) there are following errors
Err :connection error: desc = "transport: Error while dialing dial tcp: lookup orderer3.example on 172.21.0.10:53: no such host". Reconnecting...
[36m2019-09-30 06:46:10.445 UTC [grpc] HandleSubConnStateChange -> DEBU 724[0m pickfirstBalancer: HandleSubConnStateChange: 0xc000488ab0, TRANSIENT_FAILURE
Err :connection error: desc = "transport: Error while dialing dial tcp: lookup orderer5.example on 172.21.0.10:53: no such host". Reconnecting...
Failed obtaining connection: could not connect to any of the endpoints: [{orderer.example:7050 [OrdererMSP]} {orderer4.example:7050 [OrdererMSP]} {orderer3.example:7050 [OrdererMSP]} {orderer2.example:7050
[OrdererMSP]} {orderer5.example:7050 [OrdererMSP]}]
[36m2019-09-30 06:46:16.785 UTC [grpc] func1 -> DEBU 761[0m Failed to dial orderer5.example:7050: context canceled; please retry.
Kindly help? @mastersingh24 @iramiller
iramiller (Mon, 30 Sep 2019 14:36:29 GMT):
@ahmad-raza does your second cluster have routing/dns entries to the first cluster? Specifically can you resolve the `ordererX.example` hostnames and route to them? When troubleshooting these types of issues I will often use `nc -z hostname.namespace 7050` which uses netcat to test for an open port ... this lets you quickly see that the dns is set and the port is accessible... the command will hang if it is not able to reach the port ... it exits immediately upon success
iramiller (Mon, 30 Sep 2019 14:37:43 GMT):
(the above netcat command being executed from a shell opened in the pod/container that appears to be having network issues reaching other parts of the network)
dsanchezseco (Mon, 30 Sep 2019 14:47:32 GMT):
Has joined the channel.
ahmad-raza (Mon, 30 Sep 2019 14:57:36 GMT):
Thanks for replying
it return orderer.example: forward host lookup failed: Unknown host . How can i made routing/dns entries to the first cluster? @iramiller
any hint?
iramiller (Mon, 30 Sep 2019 15:06:23 GMT):
you will need to build a network route between the clusters if you wish to resolve those entries... you could create a Kubernetes service with a custom endpoint that directs `ordererX.namespace` to an external IP address of the orderer if you source cluster has that orderer exposed to the internet
ahmad-raza (Mon, 30 Sep 2019 19:59:27 GMT):
exposing orderer through service type "NodePort" to the internet. is it a good practice??
ahmad-raza (Mon, 30 Sep 2019 20:02:10 GMT):
although i am using nodeport type services to expose all peers CAs and orderers
iramiller (Mon, 30 Sep 2019 21:02:15 GMT):
For my environments there is no reason that these services would ever be invoked by entities outside of our consortium -- therefore all of ours are only exposed on an internal network.
iramiller (Mon, 30 Sep 2019 21:02:51 GMT):
You may wish to explore either making a private network between your nodes and/or using mutual TLS to further harden your configuration.
iramiller (Mon, 30 Sep 2019 21:04:45 GMT):
In either case above you can use kubernetes services to assist with the name resolution requirements. Some further reading here: https://kubernetes.io/docs/concepts/services-networking/service/#externalname
sandy (Tue, 01 Oct 2019 10:02:42 GMT):
Has joined the channel.
vanitas92 (Tue, 01 Oct 2019 15:13:27 GMT):
Im considering using Hashicorp Vault for storing safely certificates and secrets in conjunction with Kubernetes. Have you managed to use vault in kubernetes and fabric? Is it really difficult? Can you share some useful guides on how to use vault for this or similar? Thank you!
iramiller (Tue, 01 Oct 2019 15:21:49 GMT):
We are using Hashicorp vault for secret storage/mangement and issuing PKI certificates for TLS and identities. (currently we are also using the Fabric CA but the intention is to remove Fabric CA entirely)
iramiller (Tue, 01 Oct 2019 15:24:47 GMT):
While I have created a direct Vault integration for Fabric the provides MSP and crypto functions we are not using it and this project is on hold pending the improvements in Fabric 2.0
iramiller (Tue, 01 Oct 2019 15:28:33 GMT):
Integrating a Vault solution into Kubernetes is an extremely complex process that should be driven by operation requirements... I don't really have a great way to help you get started with that beyond saying that you should expect to do a bunch of Google research and reading. The Fabric Kubernetes channel is not really the best place to dive into this either.
vanitas92 (Tue, 01 Oct 2019 15:30:25 GMT):
Thank you for your responses! Indeed it is not the best place to deep dive Vault here but was interested on your experience. I have not seen any updates or improvement announces regarding Fabric CA in 2.0, what are they planning to feature in the new release?
iramiller (Tue, 01 Oct 2019 15:32:47 GMT):
The updates I am interested in are the improved source code structure and focus on breaking up the monolithic source in such a way as to better support plugins. The current setup does not lend itself well to replacing functions with external modules that are build separately. Until the core project actually starts building a system using a modular approach maintaining a fork is really the only way to replace/customize these major components
iramiller (Tue, 01 Oct 2019 15:34:31 GMT):
Having implemented a replacement for the MSP system that uses vault the determination was made that this required far too many deep changes in the source tree to maintain in our fork long term... the cost/benefit/overhead of integrating upstream changes just isn't there.
dtomczyk (Fri, 04 Oct 2019 16:25:13 GMT):
Has joined the channel.
skyfan (Mon, 07 Oct 2019 12:47:25 GMT):
Has joined the channel.
adityanalge (Tue, 08 Oct 2019 00:03:50 GMT):
This might be a silly question, but I am having some trouble creating secrets/configmaps for secret keys as the key names are not deterministic. Any hints/ideas?
iramiller (Tue, 08 Oct 2019 14:16:42 GMT):
are you creating these from a helm chart or similar?
iramiller (Tue, 08 Oct 2019 14:19:31 GMT):
```
{{- range $k, $v := untilStep 0 (int (.Values.peer.replicas)) 1 }}
apiVersion: v1
kind: Secret
metadata:
name: peer-{{$v}}-msp
{{- end }}
```
iramiller (Tue, 08 Oct 2019 14:20:46 GMT):
above portion of a helm chart that uses the `.Values.peer.replicas` setting (typically 3) to iteratively create a set of secrets... this chart is using the stateful set concept in Kubernetes for peers...
iramiller (Wed, 09 Oct 2019 14:47:21 GMT):
I would like to re-evaluate replacing/augmenting the dockercontroller in Fabric 2.0 with my custom kubernetescontroller given the many updates in master for the container runtime and build abstractions that are included. For reference this controller provides direct kubernetes scheduling of chain code containers for proper cloud native deployment and orchestration. We have successfully used this extension in several dozen Kubernetes clusters on various public clouds. Ideally this effort could be repackaged into a plugin without a fork. The current peer serve method is a whopping 600+ lines of hard coded initialization wrappers around the docker VM. I see that @jyellick is one of the primary developers in that area of the code base. Perhaps @jyellick or someone else could open a discussion with me directly or in this channel to discuss the best way to do this? Perhaps the time isn't quite right due to in progress refactoring?
jyellick (Wed, 09 Oct 2019 14:47:21 GMT):
Has joined the channel.
jyellick (Thu, 10 Oct 2019 18:25:57 GMT):
@iramiller I'd suggest that you look at the 'external builders' proposal from the contributor calls or in JIRA https://jira.hyperledger.org/browse/FAB-13584
jyellick (Thu, 10 Oct 2019 18:26:27 GMT):
This separates the packaging/build from the core peer functions, and allows arbitrary implementations, such as one based on k8s to be implemented in userspace.
iramiller (Thu, 10 Oct 2019 19:29:34 GMT):
@jyellick I am familiar with the FAB-13584 proposal and FAB-13582... I am reflecting on those initiatives as well as what I see in the codebase currently... neither of those tickets have recent updates/status reports reflecting the state of code changes that are in progress. This is why I am asking about how stable the break down is based on what is currently in the peer serve method specifically. If a high rate of churn is still expected for the 2.0 release then that is all I need to know for now. Conversely if the code base is considered to be stabilizing then I will have more specific questions to ask as I look into it further.
adityanalge (Fri, 11 Oct 2019 01:00:46 GMT):
How are you handling backing up peers and orderers? I understand that you have to back up the production folder and certificates. However, can the production folder for peer0.org1.example.com be used to bring up peer1.org1.example.com?
adityanalge (Fri, 11 Oct 2019 02:23:28 GMT):
Also, do all 5 orderers in RAFT have to be backed up or backing up one is enough?
adityanalge (Fri, 11 Oct 2019 02:23:28 GMT):
Also, do all 5 orderers' production in RAFT have to be backed up or backing up one production folder is enough?
jyellick (Fri, 11 Oct 2019 14:14:02 GMT):
The code is mostly present and should be fairly stable at this point, but, probably best to wait for the documentation which should become available in the next few weeks.
iramiller (Fri, 11 Oct 2019 14:21:52 GMT):
:thumbsup:
Randyshu2018 (Mon, 14 Oct 2019 05:36:31 GMT):
Has joined the channel.
DrTES (Thu, 17 Oct 2019 17:01:30 GMT):
Has joined the channel.
Shubham-koli (Fri, 18 Oct 2019 04:50:48 GMT):
Has joined the channel.
balazsprehoda (Thu, 24 Oct 2019 10:21:07 GMT):
Has joined the channel.
rmscott (Thu, 24 Oct 2019 15:40:56 GMT):
Has joined the channel.
shivraj (Sat, 26 Oct 2019 12:37:57 GMT):
Has joined the channel.
shivraj (Sat, 26 Oct 2019 12:38:02 GMT):
@iramiller , @yacovm and all are open to answer , Guys is it possible to scale peer or ca pod to make it highly available service ? if not why?
shivraj (Sat, 26 Oct 2019 14:19:32 GMT):
@iramiller in the sense, handling huge loads...
yacovm (Sun, 27 Oct 2019 14:52:29 GMT):
@shivraj each peer needs to have its own certificate. As for CA - I guess it's possible to scale it, but I would still give each CA its own key, for security reasons.
shivraj (Mon, 28 Oct 2019 08:29:15 GMT):
@yacovm @iramiller is there any way to create a setup where there is logical grouping among peers pods so there is a load distribution happening on top of it ?
iramiller (Mon, 28 Oct 2019 14:23:28 GMT):
@shivraj ... have you done any testing to determine how your infrastructure responds to your workloads? Have you considered the structure for your chaincode (both the code itself as well as the data structures and the balance between what is on chain and what is stored off of it? A read heavy workload will have different infrastructure requirements than a write heavy workload ... The amount of read/write contention in your workload will be another significant consideration. The latency in your network between orderer nodes as well as from orderers to peers and intra-peer gossip are all critical. If the distribution of committed blocks is slow then you are prone to MVCC errors. The CA pod has essentially no load on it in a production network. Your question for how to scale this pod leads me to believe that you may need to do some significant systems architecture research and testing before you pursue scaling the peer workloads further.
iramiller (Mon, 28 Oct 2019 14:24:54 GMT):
@shivraj in short if you have a more tightly scoped question regarding scaling with specific directions you are looking to increase capacity or throughput in then the members of this forum can likely provide a more helpful answer.
AntonyZanetti (Tue, 29 Oct 2019 14:49:42 GMT):
Has joined the channel.
shivraj (Wed, 30 Oct 2019 04:32:02 GMT):
@yacovm @iramiller thanks for suggestions
adarshaJha (Thu, 31 Oct 2019 11:16:47 GMT):
Has joined the channel.
adarshaJha (Thu, 31 Oct 2019 11:16:48 GMT):
is there any tutorial to provide basic understanding of how to use kubernetes in HLF ?
iramiller (Thu, 31 Oct 2019 15:00:26 GMT):
that is a very broad question... one starting point for you would be to review the Helm charts for HLF (https://github.com/helm/charts/tree/master/stable/hlf-peer) and examine how those were implemented.
sureshtedla (Mon, 04 Nov 2019 14:06:02 GMT):
Hi All, How to deploy hyperledger fabric network on kubernetes
sureshtedla (Mon, 04 Nov 2019 14:06:21 GMT):
can anyone share the docs for the same
AliciaKiran (Wed, 06 Nov 2019 12:28:29 GMT):
Has joined the channel.
delao (Thu, 07 Nov 2019 20:26:27 GMT):
Hello everyone, how are you? I have a question for you that have been bugging me for a few days now: I have a free IBM Kubernetes Service cluster running, and I want to run some fabric containers there. My problem is: IKS doesn't use docker anymore, as it now uses containerd. how should I set the CORE_VM_ENDPOINT for the peers if I don't have a docker.sock available?
sureshtedla (Fri, 08 Nov 2019 13:00:10 GMT):
Hi All,
sureshtedla (Fri, 08 Nov 2019 13:00:42 GMT):
I have one master node and two worker nodes If Master node fails or shutdown
sureshtedla (Fri, 08 Nov 2019 13:01:00 GMT):
How to start it(Master Node) again
sureshtedla (Fri, 08 Nov 2019 13:01:29 GMT):
If worker node down and how to start it again
rpocase (Tue, 12 Nov 2019 15:03:50 GMT):
Has joined the channel.
rpocase (Tue, 12 Nov 2019 15:03:51 GMT):
Hey everyone - I'm trying to get a gauge for how people are actually dealing with crypto management in production. The best strategy I've been able to come up with is manually deploying specific secrets for tls-ca/ca, then every thing else being dynamic through initContainers on peers/orderers/etc against your organizations existing secrets. All shared organization crypto identities could be shared through secret volumes with nodes being restarted as needed when new identities become available
I haven't seen any kubernetes examples that aren't using toy crypto material. Service discovery seems to solve part of the problem for client applications, but you still wind up with some amount of hardcoded assumptions around core peers
DilipManjunatha (Wed, 13 Nov 2019 13:08:55 GMT):
Has joined the channel.
rpocase (Wed, 13 Nov 2019 15:56:39 GMT):
Looks like my proposed solution is basically what the WIP blockchain-automation-framework lab is doing. Generate crypto pseudo-manually through fabric-ca CLIs and upload to vault. Use initContainers to distribute relevant crypto material before launching peers. This seems finish if peer growth rate is relatively low. Hooking in client applications can still use service discovery to reduce their overhead, so backbone infrastructure remains the biggest pain point
mbanerjee (Fri, 15 Nov 2019 04:04:26 GMT):
What are the memory requirements to run fabirc (2 org network with raft consensus)? Any guidance or performance metrics will be helpful?
mbanerjee (Fri, 15 Nov 2019 04:04:26 GMT):
What are the memory requirements to run fabirc (2 org network with raft consensus)? Any guidance or performance metrics will be helpful.
AndresMartinezMelgar.itcl (Fri, 15 Nov 2019 07:29:39 GMT):
Hi. i have hyperledger fabric network inside a kubernetes cluster. All of this inside a localhost(192.168.0.X)
My ask is the next: how can i bring it to a cluster in several network locations?
All examples i see are in same network. Thx anyway
jyxie2007 (Mon, 18 Nov 2019 03:59:58 GMT):
Has joined the channel.
delao (Tue, 19 Nov 2019 16:56:08 GMT):
Hello people, I am facing a problem and I hope you guys can help me. I'm trying to communicate a Orderer node running on IBM Kubernetes Services with a peer running on docker on my machine. When I try to create a new channel, I am getting a TLS handshake failed and on my peer I can see this: `2019-11-19 16:55:35.867 UTC [grpc] createTransport -> DEBU 03f grpc: addrConn.createTransport failed to connect to {
delao (Tue, 19 Nov 2019 16:56:59 GMT):
I have created my certificates with either cryptogen and fabric ca
delao (Tue, 19 Nov 2019 16:56:59 GMT):
I have created my certificates with either cryptogen and fabric ca and the outcome is the same
iramiller (Wed, 20 Nov 2019 19:53:53 GMT):
you need to use a DNS name that matches one of the SANS on the certificate used ... (or an IP address entry on the certificate as the error message indicates which I would not recommend unless that address is not expected to change [ever])
iramiller (Wed, 20 Nov 2019 19:55:05 GMT):
inside of Kubernetes you can leverage custom services to create a DNS resolution if this is for an outbound connection... on your localhost a hosts file entry can be used for development name resolution...
sung (Thu, 21 Nov 2019 08:48:21 GMT):
Has joined the channel.
Esegarra (Fri, 22 Nov 2019 12:41:28 GMT):
Has joined the channel.
Esegarra (Tue, 26 Nov 2019 14:11:15 GMT):
Hello!
I am working on raising a Hyperledger Fabric network production ready on kubernetes. I use a pod with Docker-in-Docker (dind) for the deployment of the chaincode. Because of this, the security team does not allow me to deploy on-premises. I have been asked to think of ways to secure it. What kind of settings can I apply to minimize the risk of using dind?
Thank you so much.
Shubham-koli (Wed, 27 Nov 2019 12:46:38 GMT):
https://github.com/APGGroeiFabriek/PIVT#what-is-this
Shubham-koli (Wed, 27 Nov 2019 12:46:51 GMT):
check this out
Esegarra (Wed, 27 Nov 2019 14:41:57 GMT):
Thanks for the information. I have seen that in this example they are mounting the Host’s Docker socket. I’m not allowed to do this kind of setting. Due to this I’m using DinD
iramiller (Wed, 27 Nov 2019 16:05:48 GMT):
@Esegarra your best bet is to fork and patch out those portions of the system with a native Kubernetes controller (that is what we did about a year ago) or more reasonably wait a few more weeks for 2.0 to come out which should make it easier to build out a proper cloud native system architecture for chaincode instances.
guptasndp10 (Thu, 28 Nov 2019 10:22:34 GMT):
Has joined the channel.
Esegarra (Thu, 28 Nov 2019 15:04:13 GMT):
@iramiller Thank you for your answer. Only three questions more… :grin: Do you have an example of this patches what you mention? Are there approximate date of the release of the version 2.0? Are there official documentation explaining the new chaincode model cloud native? I have read the new lifecycle but nothing about this cloud approach. Thanks!!
iramiller (Fri, 29 Nov 2019 16:58:33 GMT):
@Esegarra here is a good starting point for your research https://jira.hyperledger.org/browse/FAB-13582
delao (Fri, 29 Nov 2019 20:09:31 GMT):
Thank you, I've appended the dns to the hosts' file and it worked
Esegarra (Mon, 02 Dec 2019 14:31:16 GMT):
Thank you so much! :thumbsup:
sandman (Tue, 03 Dec 2019 09:01:49 GMT):
Has joined the channel.
sandman (Tue, 03 Dec 2019 09:05:28 GMT):
Hello, my chaincode instantiation operation is failing, error in peer logs is:```
2019-12-03 08:49:40.672 UTC [endorser] callChaincode -> INFO 067 [customchannel][af093dc4] Entry chaincode: name:"lscc"
2019-12-03 08:49:40.680 UTC [endorser] callChaincode -> INFO 068 [customchannel][af093dc4] Exit chaincode: name:"lscc" (7ms)
2019-12-03 08:49:40.680 UTC [endorser] ProcessProposal -> ERRO 069 [customchannel][af093dc4] simulateProposal() resulted in chaincode name:"lscc" response status 500 for txid: af093dc45ba6f499e1c7e474225f59f3b0677858146603dda458a1c59f07cd9d
```
sandman (Tue, 03 Dec 2019 09:05:28 GMT):
Hello, my chaincode instantiation operation is failing, error in peer logs is:```
2019-12-03 08:49:40.672 UTC [endorser] callChaincode -> INFO 067 [customchannel][af093dc4] Entry chaincode: name:"lscc"
2019-12-03 08:49:40.680 UTC [endorser] callChaincode -> INFO 068 [customchannel][af093dc4] Exit chaincode: name:"lscc" (7ms)
2019-12-03 08:49:40.680 UTC [endorser] ProcessProposal -> ERRO 069 [customchannel][af093dc4] simulateProposal() resulted in chaincode name:"lscc" response status 500 for txid: af093dc45ba6f499e1c7e474225f59f3b0677858146603dda458a1c59f07cd9d
```
I think its an error with peer env CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE but im not sure, any poniters are appereciated.
razasikander (Tue, 03 Dec 2019 11:39:43 GMT):
Has joined the channel.
razasikander (Tue, 03 Dec 2019 11:39:47 GMT):
Hello how have you
razasikander (Tue, 03 Dec 2019 11:40:16 GMT):
appended the dns
laues (Tue, 03 Dec 2019 15:10:04 GMT):
Has joined the channel.
iramiller (Tue, 03 Dec 2019 15:56:03 GMT):
The advice above is to append the DNS entry to the hosts file. https://en.wikipedia.org/wiki/Hosts_(file)
iramiller (Tue, 03 Dec 2019 15:57:22 GMT):
Within Kubernetes a similar solution is handled using a Headless service without a selector. https://kubernetes.io/docs/concepts/services-networking/service/#headless-services
iramiller (Tue, 03 Dec 2019 15:57:22 GMT):
Within Kubernetes a similar solution is handled using a Headless service without a selector. https://kubernetes.io/docs/concepts/services-networking/service/#externalname
razasikander (Wed, 04 Dec 2019 04:37:08 GMT):
apiVersion: v1
kind: Service
metadata:
annotations:
kompose.cmd: kompose -f docker-compose_PoE.yaml convert --volumes hostPath
kompose.version: 1.16.0 (0c01309)
creationTimestamp: null
labels:
io.kompose.service: peer0-org1-in
name: peer0-org1-in
spec:
ports:
- name: "2051"
port: 2051
targetPort: 7051
- name: "2053"
port: 2053
targetPort: 7053
selector:
io.kompose.service: peer0-org1-in
type: ExternalName
externalName: peer0.org1.in
status:
loadBalancer: {}
razasikander (Wed, 04 Dec 2019 04:37:20 GMT):
this is my service file
razasikander (Wed, 04 Dec 2019 04:37:56 GMT):
it say sError: error getting endorser client for channel: endorser client failed to connect to peer0.org.in:7051: failed to create new connection: context deadline exceeded
razasikander (Wed, 04 Dec 2019 04:46:39 GMT):
when i wget it says Resolving peer0.org1.in (peer0.org1.in)... failed: Temporary failure in name resolution.
lionelronaldo (Wed, 04 Dec 2019 09:10:44 GMT):
Has joined the channel.
iramiller (Wed, 04 Dec 2019 15:31:39 GMT):
@razasikander you should be able to list services in the org1 namespace of your cluster and see peer0 listed ... additionally you may wish to start a container with dns tools in your cluster in the same namespace as your peer pods/containers for troubleshooting DNS resolution.
iramiller (Wed, 04 Dec 2019 15:35:19 GMT):
what I believe you will see based on the configuration you have above is a service with the name `peer0-org1-in` ... this name will not resolve as you have intended unless this name is what you are using in your configuration (compared to the dot separated version)
iramiller (Wed, 04 Dec 2019 15:36:16 GMT):
additionally the external name reference requires an external valid dns name ... and `org.in` is not an externally valid dns name ...
vinayakkumar (Thu, 05 Dec 2019 09:50:41 GMT):
Has left the channel.
sandman (Thu, 05 Dec 2019 10:33:17 GMT):
I facing errors while instantiating chaincode, I havent specified any policies in my configtx and since opeartion upto this stage ie - channel create/join, chaincode install worked without errors so I assume its not an error with certificates as well. Any pointers?
iramiller (Thu, 05 Dec 2019 19:58:00 GMT):
ou will need to give us more to go on than "facing errors" ... which errors? What kind of setup are you using, etc.
iramiller (Thu, 05 Dec 2019 19:58:00 GMT):
you will need to give us more to go on than "facing errors" ... which errors? What kind of setup are you using, etc.
sandman (Fri, 06 Dec 2019 04:46:32 GMT):
The setup is for 2 orgs (buyer, lender) nad solo orderer(in seller org) . Here are the logs of my peer pertaing to chaincode instatiation '''
sandman (Fri, 06 Dec 2019 04:56:12 GMT):
sandman - Fri Dec 06 2019 10:25:51 GMT+0530 (India Standard Time).txt
karthiknvlr (Fri, 06 Dec 2019 05:55:25 GMT):
Has joined the channel.
iramiller (Fri, 06 Dec 2019 15:59:24 GMT):
based on the output in your log file this isn't a Kubernetes issue... it appears that your request is not signed by a certificate which is part of the expected MSP. ... The lscc is looking for buyer/seller MSP members but the request was submitted with a lender MSP signature.
sandman (Mon, 09 Dec 2019 07:43:13 GMT):
Thanks for the tip.
sandman (Mon, 09 Dec 2019 07:44:07 GMT):
after solving this, I'm getting "Error: could not assemble transaction, err proposal response was not successful, error code 500, msg error starting container: error starting container: invalid endpoint
" error.
sandman (Mon, 09 Dec 2019 07:45:02 GMT):
logs on peer are ```
^[[36m2019-12-09 07:36:18.365 UTC [chaincode] Start -> DEBU a58^[[0m start container with args: chaincode -peer.address=peer1.lender.routingtest34.zeeve.net:7052
^[[36m2019-12-09 07:36:18.365 UTC [chaincode] Start -> DEBU a59^[[0m start container with env:
CORE_CHAINCODE_LOGGING_LEVEL=info
CORE_CHAINCODE_LOGGING_SHIM=warning
CORE_CHAINCODE_LOGGING_FORMAT=%{color}%{time:2006-01-02 15:04:05.000 MST} [%{module}] %{shortfunc} -> %{level:.4s} %{id:03x}%{color:reset} %{message}
CORE_CHAINCODE_ID_NAME=mycc:1.0
CORE_PEER_TLS_ENABLED=true
CORE_TLS_CLIENT_KEY_PATH=/etc/hyperledger/fabric/client.key
CORE_TLS_CLIENT_CERT_PATH=/etc/hyperledger/fabric/client.crt
CORE_PEER_TLS_ROOTCERT_FILE=/etc/hyperledger/fabric/peer.crt
^[[36m2019-12-09 07:36:18.365 UTC [container] lockContainer -> DEBU a5a^[[0m waiting for container(mycc-1.0) lock
^[[36m2019-12-09 07:36:18.365 UTC [container] lockContainer -> DEBU a5b^[[0m got container (mycc-1.0) lock
^[[36m2019-12-09 07:36:18.365 UTC [dockercontroller] Start -> DEBU a5c^[[0m failed to get docker client%!(EXTRA string=error, *errors.errorString=invalid endpoint) imageName=dev-peer1.lender.routingtest34.zeeve.net-mycc-1.0-020a6739cc10449c90d2444dd51c2accf233006c98ec077503348af3c110ff92 containerName=dev-peer1.lender.routingtest34.zeeve.net-mycc-1.0
^[[36m2019-12-09 07:36:18.365 UTC [container] unlockContainer -> DEBU a5d^[[0m container lock deleted(mycc-1.0)
^[[36m2019-12-09 07:36:18.365 UTC [chaincode] Launch -> DEBU a5e^[[0m stopping due to error while launching: invalid endpoint
error starting container
error starting container
^[[36m2019-12-09 07:36:18.365 UTC [container] lockContainer -> DEBU a5f^[[0m waiting for container(mycc-1.0) lock
^[[36m2019-12-09 07:36:18.365 UTC [container] lockContainer -> DEBU a60^[[0m got container (mycc-1.0) lock
^[[36m2019-12-09 07:36:18.365 UTC [dockercontroller] Stop -> DEBU a61^[[0m stop - cannot create client invalid endpoint
^[[36m2019-12-09 07:36:18.365 UTC [container] unlockContainer -> DEBU a62^[[0m container lock deleted(mycc-1.0)
^[[36m2019-12-09 07:36:18.366 UTC [chaincode] Launch -> DEBU a63^[[0m stop failed: invalid endpoint
error stopping container
^[[36m2019-12-09 07:36:18.366 UTC [chaincode] Launch -> DEBU a64^[[0m launch complete
^[[36m2019-12-09 07:36:18.366 UTC [chaincode] Deregister -> DEBU a65^[[0m deregister handler: mycc:1.0
^[[34m2019-12-09 07:36:18.366 UTC [endorser] callChaincode -> INFO a66^[[0m [customchannel][6c56fa77] Exit chaincode: name:"lscc" (10ms)
^[[31m2019-12-09 07:36:18.366 UTC [endorser] SimulateProposal -> ERRO a67^[[0m [customchannel][6c56fa77] failed to invoke chaincode name:"lscc" , error: invalid endpoint
error starting container
error starting container
^[[36m2019-12-09 07:36:18.366 UTC [endorser] SimulateProposal -> DEBU a68^[[0m [customchannel][6c56fa77] Exit
^[[36m2019-12-09 07:36:18.366 UTC [lockbasedtxmgr] Done -> DEBU a69^[[0m Done with transaction simulation / query execution [6c56fa774ef3be9b30d82466b5b2e6cb594182c6d2e2f47adabc739a8e500d0c]
^[[36m2019-12-09 07:36:18.366 UTC [endorser] func1 -> DEBU a6a^[[0m Exit: request from 10.244.1.105:57648
^
```
sandman (Mon, 09 Dec 2019 09:32:49 GMT):
invoking "curl --unix-socket /host/var/run/docker.sock http://localhost/events" inside peer container Im able to get responses. So I assume, peer has the correct value of "CORE_VM_ENDPOINT".
iramiller (Mon, 09 Dec 2019 15:54:01 GMT):
debugging these kinds of errors can be quite difficult ... based on what you have in your log I would look at the DNS resolution _inside_ your chain code container... by default the DNS resolution inside this container will not know the proper address of the peer it is attempting to talk back to.
iramiller (Mon, 09 Dec 2019 15:54:51 GMT):
Verify that the hostname `chaincode -peer.address=peer1.lender.routingtest34.zeeve.net:7052` resolves to the correct interface inside the chaincode container so that it connects to the host peer.
iramiller (Mon, 09 Dec 2019 15:57:36 GMT):
(this privileged docker inside kubernetes thing is the worst part of Hyperledger Fabric in the 1.x series and reflects the lack of systems engineering for cloud based environments, the 2.x version has extensive rework of the peer/chaincode architecture to address these issues.. hopefully we can start building on the 2.0 series in the next few months)
iramiller (Mon, 09 Dec 2019 15:58:54 GMT):
(in the mean time you will encounter some challenges getting this to work and without a fork that properly schedules kubernetes workloads your security architecture will be compromised, but know that others on here have persevered and deployed production systems of Fabric in Kubernetes on the 1.x branch)
sandman (Tue, 10 Dec 2019 01:53:15 GMT):
I have tried changing this to service name. Im able to nc this port and service name combination. It is still not working.
sandman (Tue, 10 Dec 2019 01:54:18 GMT):
What is the need for docker in docker if peer has access to docker.sock file?
CodeReaper (Tue, 10 Dec 2019 07:26:22 GMT):
Has joined the channel.
iramiller (Tue, 10 Dec 2019 18:22:48 GMT):
@sandman ... the issue here is that you are thinking about network connectivity from the pod perspective and cont from a container attached to the node itself (outside of kubernetes) which is how the chaincode container will be scheduled when directly interacting with the docker.sock endpoint. You need to modify the DNS settings on the docker host VM itself to make this work. If you are on the pod itself when the chaincode request comes in... and you have a container running with the docker tools installed, and a mount of the host docker sock ... then you can use the docker tools to inspect the running chaincode container (attaching to it even) and run the DNS lookups from there...
iramiller (Tue, 10 Dec 2019 18:23:43 GMT):
you will find that within that context your DNS resolution will not work UNLESS you have updated the DNS settings that docker is using in that context to point to the kube-dns servers... and also included the appropriate DNS suffix search orders
sandman (Wed, 11 Dec 2019 06:03:19 GMT):
oh I see, Thanks
mrudav.shukla (Mon, 16 Dec 2019 08:12:31 GMT):
Has joined the channel.
mrudav.shukla (Mon, 16 Dec 2019 08:17:03 GMT):
I am not able to instantiate chaincode from the peer. It would be great if some one can guide me as I'm pretty new to using Kubernetes. I am not able to figure out whether to create a separate container to instantiate the chaincode or the peer could do it on its own. Also, I'm confused on using docker.sock for instantiation. I'm using eks and have simple setup of one peer and one orderer. Peer is able to create and join the channel. Peer is able to install the chaincode however, peer is not able to instantiate the chaincode. Stack Overflow: https://stackoverflow.com/questions/59288613/chaincode-instantiation-fails-in-aws-eks-network
vanitas92 (Mon, 16 Dec 2019 15:09:26 GMT):
Has anyone tested the EXTERNALBUILDERS feature for chaincode in 2.0.0 beta release?
vanitas92 (Mon, 16 Dec 2019 15:39:09 GMT):
From the code shared in stackoverflow, you are mounting the docker socket into the peer container, now this implementation could not be working depending on the AMI you are using for your EKS nodes. I would suggest using the Docker-indocker approach as it would be difficult to know when your chaincodes are up and and running or down. You can find more info here: https://medium.com/kokster/simpler-setup-for-hyperledger-fabric-on-kubernetes-using-docker-in-docker-8346f70fbe80
mrudav.shukla (Mon, 16 Dec 2019 17:39:08 GMT):
Sure. Let me check this.
mrudav.shukla (Tue, 17 Dec 2019 11:16:25 GMT):
Hi Vanitas, I tried this Docker in Docker approach and the previous error is now gone. However, I'm still facing issues instantiating the chaincode.
mrudav.shukla (Tue, 17 Dec 2019 11:16:42 GMT):
I've made updates in the stackoverflow question: https://stackoverflow.com/questions/59288613/chaincode-instantiation-fails-in-aws-eks-network
mrudav.shukla (Tue, 17 Dec 2019 11:17:12 GMT):
If we use dind approach, do we still need to mount docker socket to host?
vanitas92 (Tue, 17 Dec 2019 12:18:03 GMT):
you do not need to mount the socket
vanitas92 (Tue, 17 Dec 2019 12:18:58 GMT):
can you share the code of your implementation? so i can further help you
mrudav.shukla (Tue, 17 Dec 2019 12:20:37 GMT):
That would be a great help. Mostly all the code is there on stackoverflow. Apart from that do you want me to share configurations related to storageclass and services?
vanitas92 (Tue, 17 Dec 2019 12:41:52 GMT):
fine for now, i just have one think different, the value of `CORE_VM_ENDPOINT` is `tcp://localhost:2375`, i have it as `http://localhost:2375`
vanitas92 (Tue, 17 Dec 2019 12:42:01 GMT):
can you try it?
mrudav.shukla (Tue, 17 Dec 2019 12:47:10 GMT):
Sure. Let me try this.
ahmad-raza (Tue, 17 Dec 2019 12:53:17 GMT):
for me when i configured dind, error is because of tls. dind:dind-stable image is tls-enabled by default. Use old dind image without tls
mrudav.shukla (Tue, 17 Dec 2019 13:05:58 GMT):
@ahmad-raza : I've disabled TLS using environment variable. I'll try out with an older one as well.
ahmad-raza (Tue, 17 Dec 2019 13:10:44 GMT):
Screenshot from 2019-12-17 18-09-37.png
ahmad-raza (Tue, 17 Dec 2019 13:10:57 GMT):
Screenshot from 2019-12-17 18-09-56.png
ahmad-raza (Tue, 17 Dec 2019 13:11:12 GMT):
Screenshot from 2019-12-17 18-10-20.png
ahmad-raza (Tue, 17 Dec 2019 13:11:41 GMT):
these would be your settings in peer container
ahmad-raza (Tue, 17 Dec 2019 13:12:34 GMT):
and these of dind container
ahmad-raza (Tue, 17 Dec 2019 13:12:47 GMT):
Screenshot from 2019-12-17 18-12-00.png
ahmad-raza (Tue, 17 Dec 2019 13:12:56 GMT):
Screenshot from 2019-12-17 18-12-14.png
mrudav.shukla (Tue, 17 Dec 2019 14:27:08 GMT):
Let me try this.
mrudav.shukla (Tue, 17 Dec 2019 17:03:12 GMT):
@ahmad-raza : Tried with the same configuration. Still the same issue.
mrudav.shukla (Tue, 17 Dec 2019 17:03:31 GMT):
@vanitas92 : Tried with http as well. Still the same issue. :(
mrudav.shukla (Tue, 17 Dec 2019 18:11:07 GMT):
@ahmad-raza : Seems the dind image version was the main issue. Tried with that image 18-dind and it worked. Thanks a lot for helping out.
mrudav.shukla (Tue, 17 Dec 2019 18:11:38 GMT):
@vanitas92 : Thanks a lot for the rightful guidance! I am able to instantiate the chaincode now.
mrudav.shukla (Tue, 17 Dec 2019 18:31:31 GMT):
Stackoverflow Answer: https://stackoverflow.com/questions/59288613/chaincode-instantiation-fails-in-aws-eks-network/59380125#59380125
vanitas92 (Tue, 17 Dec 2019 19:45:57 GMT):
oh i have the `stable-dind` version of image and it is working, might have the old version still using, thanks for the hint!
AbhijeetSamanta (Thu, 19 Dec 2019 04:35:54 GMT):
Has joined the channel.
Khaled.MH (Tue, 24 Dec 2019 10:39:41 GMT):
Hello guys
Khaled.MH (Tue, 24 Dec 2019 10:40:05 GMT):
how can i implement service discovery so peer will be able to expose these type of information to the clients ?
AbhijeetSamanta (Tue, 24 Dec 2019 19:57:58 GMT):
Hi All, I am trying to setup on HLF on EKS. I had setup till generate artefact and cryto material. also raft orderer running, but when I am creating the peers its give error as " Could not connect to Endpoint: peer1-org1-service:7051, InternalEndpoint: peer1-org1-service:7051, PKI-ID:
mrudav.shukla (Wed, 25 Dec 2019 05:44:25 GMT):
What type of service are you using?
kirikiri (Thu, 26 Dec 2019 14:23:22 GMT):
Has joined the channel.
kirikiri (Thu, 26 Dec 2019 14:24:42 GMT):
Hello, is there any full guide of setup fabric in k8s with 1 org and chaincode and wallets?
iramiller (Thu, 26 Dec 2019 22:07:45 GMT):
if anyone else was reading through this and considering DinD for Kubernetes ... an alternative approach (if you have go experience) would be to build upon my work for a Kubernetes controller that schedules chaincode containers properly (until 2.0 comes out and we can rebuild into a nice add on module) https://github.com/FigureTechnologies/fabric/blob/release-1.4/core/container/kubernetescontroller/kubernetescontroller.go
iramiller (Thu, 26 Dec 2019 22:08:58 GMT):
note that the above approach isn't plug and play ... and it relies our our approach of releasing chaincode containers built by our Jenkins pipeline internally ... but any external chaincode build process could do the same.
AbhijeetSamanta (Fri, 27 Dec 2019 06:47:13 GMT):
Sorry to bother all I have fixed the issue thanks for reply
kirikiri (Mon, 30 Dec 2019 12:56:14 GMT):
Hello, could anybody help with some example of how to enable ingress for the peer? As I found in docs nginx-ingress can proxy http2 or grpc requests only with enabled tls, but I cannot understand where should I get certificates and how to pass them to the ingress
tengc (Tue, 31 Dec 2019 03:28:21 GMT):
Has joined the channel.
tengc (Tue, 31 Dec 2019 03:28:21 GMT):
I noticed when setting up k8s that there are quite a number of ports that must be made accessible in order for the cluster to function. For setups that span multiple locations, is it necessary to split the network into localized clusters?
kirikiri (Thu, 02 Jan 2020 09:52:39 GMT):
Hello, how to setup nginx ingress for peers and orderers to be able to submit transactions outside the cluster?
AbhijeetSamanta (Sun, 05 Jan 2020 08:39:24 GMT):
Hi all anybody have implement the node sdk with k8s.
roclee (Mon, 06 Jan 2020 03:03:29 GMT):
Has joined the channel.
AbhijeetSamanta (Tue, 07 Jan 2020 07:52:56 GMT):
Hi All, I am stuck in one issue with connection profile. Let me tell what I am trying to do. I have created HL network on AWS EKS with on ca-root and 2 ca-client. I have 2 organisations which have 2 peer each. I am using the raft orderering sevice with 3 orderers. all the network running fine now I want to connect it with node applications API's for that I have created the connection profile, but it not connected I am getting issue as "2020-01-07T07:36:07.526Z - error: [FabricCAClientService.js]: Failed to enroll admin, error:%o message=Calling enrollment endpoint failed with error [Error: connect ETIMEDOUT 10.100.81.135:7054], stack=Error: Calling enrollment endpoint failed with error [Error: connect ETIMEDOUT 10.100.81.135:7054]
"
AbhijeetSamanta (Tue, 07 Jan 2020 07:52:56 GMT):
Hi All, I am stuck in one issue with connection profile. Let me tell what I am trying to do. I have created HL network on AWS EKS with on ca-root and 2 ca-client. I have 2 organisations which have 2 peer each. I am using the raft orderering sevice with 3 orderers. all the network running fine now I want to connect it with node applications API's for that I have created the connection profile, but it not connected I am getting issue as `"2020-01-07T07:36:07.526Z - error: [FabricCAClientService.js]: Failed to enroll admin, error:%o message=Calling enrollment endpoint failed with error [Error: connect ETIMEDOUT 10.100.81.135:7054], stack=Error: Calling enrollment endpoint failed with error [Error: connect ETIMEDOUT 10.100.81.135:7054]
" `
AbhijeetSamanta (Tue, 07 Jan 2020 07:52:56 GMT):
Hi All, I am stuck in one issue with connection profile. Let me tell what I am trying to do. I have created HL network on AWS EKS with on ca-root and 2 ca-client. I have 2 organisations which have 2 peer each. I am using the raft orderering sevice with 3 orderers. all the network running fine now I want to connect it with node applications API's for that I have created the connection profile, but it not connected I am getting issue as
`"2020-01-07T07:36:07.526Z - error: [FabricCAClientService.js]: Failed to enroll admin, error:%o message=Calling enrollment endpoint failed with error [Error: connect ETIMEDOUT 10.100.81.135:7054], stack=Error: Calling enrollment endpoint failed with error [Error: connect ETIMEDOUT 10.100.81.135:7054]
``` `
" ````
anybody have any experience how to do it. I have tried all cased like loadbalance, clusterIP etc.
```
AbhijeetSamanta (Tue, 07 Jan 2020 07:52:56 GMT):
Hi All, I am stuck in one issue with connection profile. Let me tell what I am trying to do. I have created HL network on AWS EKS with on ca-root and 2 ca-client. I have 2 organisations which have 2 peer each. I am using the raft orderering sevice with 3 orderers. all the network running fine now I want to connect it with node applications API's for that I have created the connection profile, but it not connected I am getting issue as
``` `
`"2020-01-07T07:36:07.526Z - error: [FabricCAClientService.js]: Failed to enroll admin, error:%o message=Calling enrollment endpoint failed with error [Error: connect ETIMEDOUT 10.100.81.135:7054], stack=Error: Calling enrollment endpoint failed with error [Error: connect ETIMEDOUT 10.100.81.135:7054] ````
anybody have any experience how to do it. I have tried all cased like loadbalance, clusterIP etc.
```
AbhijeetSamanta (Tue, 07 Jan 2020 07:52:56 GMT):
Hi All, I am stuck in one issue with connection profile. Let me tell what I am trying to do. I have created HL network on AWS EKS with on ca-root and 2 ca-client. I have 2 organisations which have 2 peer each. I am using the raft orderering sevice with 3 orderers. all the network running fine now I want to connect it with node applications API's for that I have created the connection profile, but it not connected I am getting issue as
``` `
`"2020-01-07T07:36:07.526Z - error: [FabricCAClientService.js]: Failed to enroll admin, error:%o message=Calling enrollment endpoint failed with error [Error: connect ETIMEDOUT 10.100.81.135:7054], stack=Error: Calling enrollment endpoint failed with error [Error: connect ETIMEDOUT 10.100.81.135:7054] ````
anybody have any experience how to do it. I have tried all cased like loadbalance, clusterIP etc.
lionelronaldo (Tue, 07 Jan 2020 09:36:06 GMT):
RAFT
iramiller (Tue, 07 Jan 2020 23:08:07 GMT):
@AbhijeetSamanta make sure you can hit your CA service from the pod you are running your code in above ... verify that your DNS name (if you are using one) resolves correctly... some times it is easier to isolate the infrastructure and make sure it is working as expected before trying to further debug the Hyperledger Fabric software
ZainabM (Thu, 09 Jan 2020 11:55:16 GMT):
Has joined the channel.
tatsu-sato (Thu, 09 Jan 2020 22:19:18 GMT):
Has joined the channel.
AbhijeetSamanta (Fri, 10 Jan 2020 06:53:15 GMT):
@iramiller Thanks for that. I have resolve the issue and it connect to my HL network however I cannot use the API which is made for register the users
AbhijeetSamanta (Fri, 10 Jan 2020 07:01:10 GMT):
I have one API which is register the user into HL network using node SDK , but it couldn't register the user. I have checked with another HL network(test) its working fine and I can register the user, however when I am trying with the k8s based network its not working and users could not register to network. Could you please help this issue. I can provide the information what you need. like connection profile file and node sdk code for user registration.
mauricio (Sat, 11 Jan 2020 15:28:09 GMT):
How service discovery works if I deploy my fabric network on Kubernetes if for default Kubernetes has a load balancer, how the SDK can discover the other peers? Should I use Kubernetes for deploy a production ready Hyperledger Fabric network?
gentios (Sun, 12 Jan 2020 20:41:16 GMT):
Hi have someone used Nephos to deploy the network in Kubernetes, I am trying to map the /crypto files to the connection-profile in order to use it with Node.js but without luck. Would appreciate if someone can help me out
gentios (Sun, 12 Jan 2020 20:47:01 GMT):
nephos
lionelronaldo (Mon, 13 Jan 2020 06:52:01 GMT):
Ex
lionelronaldo (Mon, 13 Jan 2020 07:20:37 GMT):
Hi there! Thanks for this forum, it already helped a lot :pray:
Now I wanted to setup Hyperledger Explorer on my cluster, and the guide by feitnomore (https://github.com/feitnomore/hyperledger-fabric-kubernetes thx at this point) uses a Network file system (NFS).
Until now, I always used K8s secrets to inject certs and other information into my Peer and Orderer pods as described in the great guides by AidTech (https://github.com/aidtechnology/hgf-k8s-workshop) and APG and Accenture NL (https://github.com/APGGroeiFabriek/PIVT).
2 questions arose:
How to distribute the crypto material securely in a production environment?
Of all three guides shared above, only AidTech doesn't use the cryptogen tool, but the actual HLF CA Container to issue keys and certificates. However, they still copy the keys from the CA Pod to the local machine, and then inject it into the Peer/Orderer Container. It doesn't seem very secure to me if private keys ever leave the pod for which they are meant for.
lionelronaldo (Mon, 13 Jan 2020 07:20:37 GMT):
Hi there! Thanks for this forum, it already helped a lot :pray:
Now I wanted to setup Hyperledger Explorer on my cluster, and the guide by feitnomore (https://github.com/feitnomore/hyperledger-fabric-kubernetes thx at this point) uses a Network file system (NFS).
Until now, I always used K8s secrets to inject certs and other information into my Peer and Orderer pods as described in the great guides by AidTech (https://github.com/aidtechnology/hgf-k8s-workshop) and APG and Accenture NL (https://github.com/APGGroeiFabriek/PIVT).
2 questions arose:
How to distribute the crypto material securely in a production environment?
Of all three guides shared above, only AidTech doesn't use the cryptogen tool, but the actual HLF CA Container to issue keys and certificates. However, they still copy the keys from the CA Pod to the local machine, and then inject it into the Peer/Orderer Container. It doesn't seem very secure to me if private keys ever leave the pod for which they are meant for.
Regarding the solution by feitnomore using NFS "Note: Crypto materials, configuration files and some scripts will be saved on this shared filesystem." : looks like all Pods can access the crypto material of other Pods, doesn't seem very secure (although I don't have any clue about NFS).
Do you know a solution to enable Hyperledger Explorer without NFS? I guess the NFS is mounted on the Hyperledger Explorer Pod because it needs some information that is inside the shared filesystem, but maybe this can be replaced somehow?
Any answers or some fruitful discussion is very much welcomed! :slight_smile:
lionelronaldo (Mon, 13 Jan 2020 07:45:06 GMT):
Hi @kirikiri. It depends on how you setup your HF network on K8s. I personally asked myself the same question last week. For the setup, I used the guide for a production environment by AidTech (https://github.com/aidtechnology/hgf-k8s-workshop). By using the helm charts hlf-peer and hlf-orderer, services for each Peer are automatically generated. Also the great guide by APG and Accenture NL (https://github.com/APGGroeiFabriek/PIVT) creates a Kubernetes service for every pod. That way you can access the Peer and Orderer Pods over their respective service :slight_smile: .
So you just need to install the nginx ingress e.g.
`helm install nginx-ingress stable/nginx-ingress`
and then create an ingress with a file ingress.yaml similar to this
`apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: peer-ingress
namespace: peers
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: peer.example.com
http:
paths:
- path: /request
backend:
serviceName: peer1-hlf-peer
servicePort: request
- path: /event
backend:
serviceName: peer1-hlf-peer
servicePort: event`
with `kubectl create -f ingress.yaml`
lionelronaldo (Mon, 13 Jan 2020 07:49:05 GMT):
In the property "serviceName" as in `serviceName: peer1-hlf-peer`you have to specify the service that corresponds to your pod.
lionelronaldo (Mon, 13 Jan 2020 08:07:06 GMT):
But as I found out, you can also assign an ingress directly in the values.yaml file for installing the helm chart. AidTech does it like this to enable an ingress for their CA Pod (https://github.com/aidtechnology/hgf-k8s-workshop/blob/master/prod_example/helm_values/ca.yaml), and in the other helm charts of Peer and Orderer there is also the possibility to inject ingress.enabled=true into the chart. This way the chart automatically creates an Ingress for your Pod :smile:
gentios (Mon, 13 Jan 2020 09:11:58 GMT):
@lionelronaldo thanks for this one man I was looking for the same
gentios (Mon, 13 Jan 2020 09:12:16 GMT):
howver Idk whether it's safe to expose the peers and orderes outside the cluster
lionelronaldo (Mon, 13 Jan 2020 10:51:31 GMT):
Yes absolutely, that's an important thought, I overlooked this until now :sweat_smile:
Currently I'm using Digital Ocean (DO) managed Kubernetes, and there you can set inbound rules for Requests. I guess an easy first precaution would be to whitelist only known IP addresses for my loadbalancer directly in DO.
However, shouldn't it be possible from anyone wanting to use the sdk to connect to my peer? The application that connects to the HF network could reside in any part of the world, shouldn't it?
This would indicate that it's safe to expose the peer outside of the cluster, because requests are anyway only executed if they come from trusted entities, i.e. parties that have a certificate that's trusted by the HF network.
Does this make sense? Or do I misunderstand something?
gentios (Mon, 13 Jan 2020 10:53:13 GMT):
No that is perfectly correct, I have the same thougts too. I am also using Digital Ocean managed Kubernetes
gentios (Mon, 13 Jan 2020 10:53:32 GMT):
Can you share the updated peer chart and order chart with ingress
lionelronaldo (Mon, 13 Jan 2020 10:54:13 GMT):
Yes, I'm trying to get it going right now. I'll share it once I found a working solution :thumbsup:
gentios (Mon, 13 Jan 2020 10:55:12 GMT):
Thank you @lionelronaldo
DollyVolley (Mon, 13 Jan 2020 15:45:50 GMT):
Has joined the channel.
adityanalge (Wed, 15 Jan 2020 18:55:16 GMT):
I had the same question @lionelronaldo. Thank you for framing it so aptly. Curious as to how more people are not talking/worried about this.
.
On a high level it is a simple problem of moving the CA root certificates from the CA pod to the Orderer/Peer pod and figuring out the most secure way to do it.
lionelronaldo (Thu, 16 Jan 2020 12:29:54 GMT):
@gentios
So i figured out how to adjust the helm charts hlf-ca, hlf-peer and hlf-orderer so that the installation of them automatically creates an ingress. :yum:
At first I wanted to use the same secret of the TLS certificate across ca, orderer and peer. So just one certifiacte for mydomain.com, and to redirect the requests to the specific services via paths like mydomain.com/ca, mydomain.com/peer and mydomain.com/orderer. Guess what, the problem was the fabric-ca-client.
When you call `FABRIC_CA_CLIENT_HOME=./config fabric-ca-client enroll -u https://ord-admin:OrdAdm1nPW@$CA_INGRESS -M ./OrdererMSP` (as in the AidTech guide https://github.com/aidtechnology/hgf-k8s-workshop/tree/master/prod_example), the $CA_INGRESS would be in our example mydomain.com/ca. Unfortunetaley the fabric-ca-client binary cuts off everthing before the last slash, and instead of https://mydomain.com/ca/enroll it always tried to call https://ca/enroll which obviously doesn't work.
I couldn't find a way around that. :disappointed: Please tell me if you know a solution that this issue!
Next try was to use a wildcard certificate for *.mydomain.com, so I only need to generate one certificate in my cluster and I don't run into the letsencrypt rate limits that easily. However, for that you need a special ClusterIssuer with a dns solver that can connect to your domain registrator. I didn't have that because I'm currently using free domains without such api service.
:feet:
So in the end, I just used the subdomains ca.mydomain.com, orderer.mydomain.com and peer.mydomain.com and generated for each one of them a TLS certificate issued by letsencrypt.
After you have a ready ClusterIssuer for the cert-manager (let's say it is "letsencrypt-production), you need to add an ingress section to each of the values.yaml files of the charts:
ca:
`ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: letsencrypt-production
path: /
hosts:
# TODO: Change this to your Domain Name
- ca.yourdomain.com
tls:
- secretName: ca--tls
hosts:
# TODO: Change this to your Domain Name
- ca.yourdomain.com`
peer:
`ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: letsencrypt-production
path: /
hosts:
# TODO: Change this to your Domain Name
- peer.yourdomain.com
tls:
- secretName: peer--tls #tls-secret
hosts:
# TODO: Change this to your Domain Name
- peer.yourdomain.com`
orderer:
`ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: letsencrypt-production
path: /
hosts:
# TODO: Change this to your Domain Name
- orderer.yourdomain.com
tls:
- secretName: orderer--tls
hosts:
# TODO: Change this to your Domain Name
- orderer.yourdomain.com`
I only included this ingress section in peer1 and ord1, as I was ok with connecting to just 1 peer and orderer. This creates 3 ingresses, which should be accessable via your domains with https. :heart_decoration:
Until now I'm not sure how to use the ingress for peer and orderer.
I guess with this setup it's possible to enable a HF network with multiple K8s clusters. :slight_smile: That will be the next step.
lionelronaldo (Thu, 16 Jan 2020 12:37:44 GMT):
Another question that arose during my work with the hlf helm charts developed by Aid:Tech:
The hlf-ca helm chart uses a requirements.lock file, which has a dependency on the postgresql 2.6.1 and mysql 0.10.2 helm charts. However these helm charts use apiVersions that are not compatible with the apiVersions of K8s 1.16 , so I cannot use them with Kubernetes 1.16 .
*Is there any possibility in K8s to override this requirements.lock file from a remote helm repo?
* :pray:
In the values.yaml for hlf-ca, I can specify the image tag of postgresql and mysql, but I cannot specify the helm chart version..
For now, I forked the repo, adjusted and ran the chart locally.
lionelronaldo (Thu, 16 Jan 2020 12:37:44 GMT):
Another question that arose during my work with the hlf helm charts developed by Aid:Tech:
The hlf-ca helm chart uses a requirements.lock file, which has a dependency on the postgresql 2.6.1 and mysql 0.10.2 helm charts. However these helm charts use apiVersions that are not compatible with the apiVersions of K8s 1.16 , so I cannot use them with Kubernetes 1.16 .
*Is there any possibility in K8s to override this requirements.lock file from a remote helm repo? * :pray:
In the values.yaml for hlf-ca, I can specify the image tag of postgresql and mysql, but I cannot specify the helm chart version..
For now, I forked the repo, adjusted and ran the chart locally.
lionelronaldo (Thu, 16 Jan 2020 12:37:44 GMT):
Another question that arose during my work with the hlf helm charts developed by Aid:Tech:
The hlf-ca helm chart uses a requirements.lock file, which has a dependency on the postgresql 2.6.1 and mysql 0.10.2 helm charts. However these helm charts use apiVersions that are not compatible with the apiVersions of K8s 1.16 , so I cannot use them with Kubernetes 1.16 .
*Is there any possibility in K8s/helm to override this requirements.lock file from a remote helm repo? * :pray:
In the values.yaml for hlf-ca, I can specify the image tag of postgresql and mysql, but I cannot specify the helm chart version..
For now, I forked the repo, adjusted and ran the chart locally.
lionelronaldo (Thu, 16 Jan 2020 12:40:50 GMT):
By the way, maybe your right that it's bad to open up your Peers and Orderers so openly to the internet. An attacker might not be able to attack via the HF network, but he might be able to attack via the K8s network. However I don't know yet a lot about K8s security.
lionelronaldo (Thu, 16 Jan 2020 12:40:50 GMT):
By the way, maybe you are right that it's bad to open up your Peers and Orderers so openly to the internet. An attacker might not be able to attack via the HF network, but he might be able to attack via the K8s network. However I don't know yet a lot about K8s security.
vanitas92 (Thu, 16 Jan 2020 14:37:39 GMT):
Has anyone tested the EXTERNALBUILDERS feature for chaincode in 2.0.0 beta release?
I'm having trouble setting up the env var CORE_CHAINCODE_EXTERNALBUILDERS in yaml as i constantly got this error: `Error: '': source data must be an array or slice, got string`
This is how i set it up:
```
- name: CORE_CHAINCODE_EXTERNALBUILDERS
value: '[{name: golang-builder, path: /builders/golang}]'
```
ZainabM (Fri, 17 Jan 2020 06:56:49 GMT):
I have not tried, but just a thought that remove quotes arround the array and try. Like `- name: CORE_CHAINCODE_EXTERNALBUILDERS
value: [{name: golang-builder, path: /builders/golang}]`
mrudav.shukla (Fri, 17 Jan 2020 08:04:45 GMT):
I have a CA container running within Pod in an amazon EKS cluster. I am exposing this using service type load balancer. It has TLS enabled. Now, since this pod is exposed using amazon’s load balancer, this load balancer will need to have an SSL cert. My question is, should this SSL cert be the certificate of the CA and be explicitly imported into amazon’s ACM or this load balancer would have a different SSL cert?
lionelronaldo (Fri, 17 Jan 2020 08:09:50 GMT):
Typically the LoadBalancer would have his own cert, not the one from the CA or some other HF component.
mrudav.shukla (Fri, 17 Jan 2020 10:58:39 GMT):
Ok. Let me try.
AbhijeetSamanta (Tue, 21 Jan 2020 07:57:22 GMT):
Hi all, I am trying to implement HLF network on k8s with aws EKS. I have some query regarding the architecture of production grade environment for the HLF network, What would be the best architecture to implement it. I am planning to create separate cluster for each org and orderer also. is it right way to implement it? also I want to know the CA architecture. Please help me in this case as I little confused with it. Thanks in advance
iramiller (Tue, 21 Jan 2020 18:42:27 GMT):
@AbhijeetSamanta if all of your clusters are managed by a single team ... perhaps relying on namespaces for separation is sufficient vs the extra infrastructure for all those extra clusters. We used ICAs (and loaded those into Fabric CAs) for each org in our network. You would want to discuss that further in the #fabric-ca channel I suspect.
lionelronaldo (Wed, 22 Jan 2020 12:07:41 GMT):
So my post about how to setup ingress gave ingresses, but I don't know how to work with them for a multicluster setup (https://chat.hyperledger.org/channel/fabric-kubernetes?msg=KcpdQiPMmHi5BqZ6Y).
Could anybody give me a hint how to use the public domain names instead of cluster specific domain names?
Is .spec.hostAliases helping for this issue? Because it only translates inner cluster domain names to ip addresses, I would need a translation to public domain names (my public LoadBalancer IP uses ingress for the different services).
:thinking:
AbhijeetSamanta (Thu, 23 Jan 2020 07:53:38 GMT):
Hi thanks for reply, yes I will ask into #fabric-ca group regarding the CA
Nammalvar (Thu, 23 Jan 2020 08:51:26 GMT):
Has joined the channel.
mrudav.shukla (Thu, 23 Jan 2020 11:52:08 GMT):
I am facing an issue similar to this: https://stackoverflow.com/questions/56735065/chaincode-is-instantiated-but-doesnt-appear-in-the-list-of-instantiated-codes. In my case, I am able to fetch the blocks using peer channel fetch command. However, though instantiation is successful, I am not able to get the list of instantiated chaincodes. Has anyone faced similar issue? I am using aws eks.
gentios (Fri, 24 Jan 2020 09:25:24 GMT):
@lionelronaldo thank you for sharing your setup, I did an initial research on this and I found some links from the at-charts of Aid:tech.
As you can see in this link: https://github.com/aidtechnology/at-charts/blob/master/hlf-peer/values.yaml#L21
In order to expose a peer we have to pass the "GRPC" annotation and if you want to use the TLS you have to pass the following config: https://github.com/aidtechnology/at-charts/blob/master/hlf-peer/values.yaml#L64 and following secrets: https://github.com/aidtechnology/at-charts/blob/master/hlf-peer/values.yaml#L84
I haven't still tested it with TLS, I am still having a problem connecting to the peer after the ingress enabled, in order to install the chaincode.
Still evaluating how to expose it correctly, if someone has setup this would be good to share some answers
gentios (Fri, 24 Jan 2020 09:26:27 GMT):
Regarding the multicluster setup, I don't have such experience yet tbh
lionelronaldo (Fri, 24 Jan 2020 10:33:48 GMT):
Great catch! I always overlooked the 'nginx.ingress.kubernetes.io/backend-protocol: "GRPC"' part!
Just tried it out without TLS and without CA in a minimalist setup of 1 peer and 1 orderer (using cryptogen).
It doesn't work just with the additional annotation of backend-protocol, but I think we are one step further.
In the peer logs there is
"External endpoint is empty, peer will not be accessible outside of its organization"
With "helm template stable/hlf-peer -n peer1 --namespace peers -f ./helm_values/peer1.yaml" I found out that the environment variables "CORE_PEER_GOSSIP_EXTERNALENDPOINT" and "CORE_PEER_GOSSIP_BOOTSTRAP" are not set, as they should be indicated by this article https://hyperledger-fabric.readthedocs.io/en/release-1.4/gossip.html .
In https://github.com/aidtechnology/at-charts/blob/b0bb623da5b864a6ffc176261909494df789f2ab/hlf-peer/values.yaml#L60 I found that it's able to set the endpoints in values.yaml. Maybe we have to set the right values there?
gentios (Fri, 24 Jan 2020 10:35:40 GMT):
@lionelronaldo no I think that is for the gossip protocol has not to do with the external url
gentios (Fri, 24 Jan 2020 10:35:56 GMT):
The thing is that if we specify the service type to LoadBalancer as here: https://github.com/aidtechnology/at-charts/blob/b0bb623da5b864a6ffc176261909494df789f2ab/hlf-peer/values.yaml#L11
gentios (Fri, 24 Jan 2020 10:36:21 GMT):
It will be exposed to public without a problem, however it's not safe for prod since the communication is not protected by TLS
lionelronaldo (Fri, 24 Jan 2020 10:48:20 GMT):
@gentios ok very interesting, thanks for the info! Guess my K8s are still not good enough.
If I enable LoadBalancer for the service instead of ClusterIP, do I still need to enable the ingress?
lionelronaldo (Fri, 24 Jan 2020 10:48:20 GMT):
@gentios ok very interesting, thanks for the info! Guess my K8s skills are still not good enough.
If I enable LoadBalancer for the service instead of ClusterIP, do I still need to enable the ingress?
lionelronaldo (Fri, 24 Jan 2020 11:00:09 GMT):
Or how do I connect orderer and peer if I exposed them via a LoadBalancer? It's ok to start without TLS. Did you ever get it to work without TLS with LoadBalancers or Ingresses?
gentios (Fri, 24 Jan 2020 11:44:36 GMT):
@lionelronaldo no haven't tried with LoadBalancer yet, but will do
gentios (Fri, 24 Jan 2020 11:44:53 GMT):
If you pass a LoadBalancer it will assign an external IP Address where you
gentios (Fri, 24 Jan 2020 11:45:07 GMT):
can connect from the fabric-sdk or however you will use it
gentios (Fri, 24 Jan 2020 11:45:14 GMT):
like: grpc://ip:address
gentios (Fri, 24 Jan 2020 11:47:22 GMT):
maybe the guys from Aid:Tech can assist us here: @nicolapaoli or @alexvicegrab
lionelronaldo (Fri, 24 Jan 2020 12:25:22 GMT):
Ok I was aware of that. However I cannot find out how to tell the orderer "connect to PEER_IP" and the peer "connect to ORDERER_IP"
lionelronaldo (Fri, 24 Jan 2020 12:26:39 GMT):
I guess only either LoadBalancer or Ingress is needed. In their prod example they set up an ingress for the CA, but leave the service with ClusterIP and do not change it to LoadBalancer.
gentios (Fri, 24 Jan 2020 12:27:01 GMT):
Yes because LoadBalancer = Ingress
gentios (Fri, 24 Jan 2020 12:27:07 GMT):
They are the same
gentios (Fri, 24 Jan 2020 12:27:19 GMT):
Ingress is a type of LoadBalancer service
lionelronaldo (Fri, 24 Jan 2020 12:27:37 GMT):
Well, an Ingress needs a LoadBalancer, but LoadBalancer can also exist without Ingress no?
gentios (Fri, 24 Jan 2020 12:28:08 GMT):
No an ingress is a type of LoadBalancer
gentios (Fri, 24 Jan 2020 12:28:19 GMT):
They are equal
gentios (Fri, 24 Jan 2020 12:28:29 GMT):
Idk too much about kubernetes too :D
gentios (Fri, 24 Jan 2020 12:29:17 GMT):
I am using nephos to deploy the HLF Kubernetes cluster
gentios (Fri, 24 Jan 2020 12:29:29 GMT):
and in their example they have a qa-tls and dev-tls
gentios (Fri, 24 Jan 2020 12:29:38 GMT):
on how to enable tls here link: https://github.com/hyperledger-labs/nephos/tree/master/examples
gentios (Fri, 24 Jan 2020 12:30:01 GMT):
I have been blind, haven't seen this :joy:
gentios (Fri, 24 Jan 2020 12:30:23 GMT):
Will share more details when I will try it out
lionelronaldo (Fri, 24 Jan 2020 12:31:24 GMT):
Aha ok. I'm not quite sure if LoaBalancer = Ingress, this article explains some differences https://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0
lionelronaldo (Fri, 24 Jan 2020 12:31:54 GMT):
Nice, I couldn't get nephos to run because I had some python issues, but good to know to you can set up a network with it!
gentios (Fri, 24 Jan 2020 12:33:39 GMT):
Yes it's easier than manual, regarding the python I guess you had a problem with the commands just use:
PYTHONPATH=. command here
lionelronaldo (Fri, 24 Jan 2020 12:37:05 GMT):
Ok maybe I'll try it in the future again, for now I wrote some nice bash scripts :slight_smile:
gentios (Fri, 24 Jan 2020 12:38:15 GMT):
ok, will give you an update if I am able to figure it out these days :D
lionelronaldo (Fri, 24 Jan 2020 13:32:16 GMT):
Ok with enabling LoadBalancers for Peer and Orderer and pointing the domain names to their IP addresses I have the same result as with ingress enabled and 'nginx.ingress.kubernetes.io/backend-protocol: "GRPC"'.
Both versions return "Error: rpc error: code = Unavailable desc = transport is closing" when trying to create a channel or do something different.
gentios (Fri, 24 Jan 2020 23:17:04 GMT):
@lionelronaldo yes I have the same error :(
gentios (Fri, 24 Jan 2020 23:19:29 GMT):
Somehow the command `peer channel list` which is exectued from inside the pod, tries to call the peer by a domain name and not like the internal kubernetes network and it's failing since it doesn't exist
lionelronaldo (Mon, 27 Jan 2020 07:03:59 GMT):
Hey @gentios Thanks a lot for your tips!! I got a minimal network running with the peer/orderer services exposed via a LoadBalancer :yum:
The next step is now to do it with the nginx-ingress controller instead, but that's more complicated. For nginx-ingress controller it's necessary to use TLS, because they only serve http/2 over the 443 port (https://github.com/kubernetes/ingress-nginx/issues/3897).
GRPC communicates over http/2.
I guess also something else needs to be adapted, because I couldn;t get it to work even with valid TLS certificates.
gentios (Mon, 27 Jan 2020 08:21:34 GMT):
@lionelronaldo did u just pust the service.Type=LoadBalancer for both peer and order and it worked ?
lionelronaldo (Mon, 27 Jan 2020 08:44:06 GMT):
Yes.
I also changed the k8s service names like peer1-hlf-peer.peers.svc.cluster.local to my actual domains like peer.toonu.com and orderer.toonu.com, In configtx.yaml, crypto-config.yaml and in my setup script.
Of course, I also changed the the paths to my crypto-material in my setup script to reflect the domains. E.g. from `MSP_DIR=./crypto-config/ordererOrganizations/orderers.svc.cluster.local/orderers/ord1-hlf-ord.orderers.svc.cluster.local/msp` to `MSP_DIR=./crypto-config/ordererOrganizations/orderer.oliveris.happyforever.com/orderers/ord1-hlf-ord.orderer.toonu.com/msp`
Then I installed my peer and orderer, and waited for the LoadBalancers to be created. Once they were created and I got their ExternalIPs, I headed over to my domain name service provider and changed the IPs of peer.toonu.com and orderer.toonu.com to their respective LoadBalancers. Then I had to wait again until the domain names actually resolve to the new IP addresses. Once they did, I could create a channel, install, instantiate and invoke an example chaincode.
lionelronaldo (Mon, 27 Jan 2020 08:44:06 GMT):
Yes.
I also changed the k8s service names like peer1-hlf-peer.peers.svc.cluster.local to my actual domains like peer.toonu.com and orderer.toonu.com, In configtx.yaml, crypto-config.yaml and in my setup script.
Of course, I also changed the the paths to my crypto-material in my setup script to reflect the domains. E.g. from `MSP_DIR=./crypto-config/ordererOrganizations/orderers.svc.cluster.local/orderers/ord1-hlf-ord.orderers.svc.cluster.local/msp` to `MSP_DIR=./crypto-config/ordererOrganizations/orderer.toonu.com/orderers/ord1-hlf-ord.orderer.toonu.com/msp`
Then I installed my peer and orderer, and waited for the LoadBalancers to be created. Once they were created and I got their ExternalIPs, I headed over to my domain name service provider and changed the IPs of peer.toonu.com and orderer.toonu.com to their respective LoadBalancers. Then I had to wait again until the domain names actually resolve to the new IP addresses. Once they did, I could create a channel, install, instantiate and invoke an example chaincode.
gentios (Mon, 27 Jan 2020 08:46:47 GMT):
Great that is perfect, with what you installed the chaincode with node sdk ?
lionelronaldo (Mon, 27 Jan 2020 08:53:15 GMT):
With `kubectl cp` I copy the chaincode into the pod, and with `kubectl exec -- bash` I install, instantaite and invoke.
Installation:
```
PEER_POD=$(kubectl get pods -n peers -l "app=hlf-peer,release=peer1" -o jsonpath="{.items[0].metadata.name}")
kubectl cp ../../../chaincodes/chaincode_example02/ peers/$PEER_POD:/var/hyperledger/production/chaincodes
kubectl exec -n peers $PEER_POD -- bash -c 'CORE_PEER_MSPCONFIGPATH=$ADMIN_MSP_PATH peer chaincode install -n mycc -v 1.0 -l node -p /var/hyperledger/production/chaincodes/chaincode_example02/node'
```
Instantiaten and invocation:
```
kubectl exec -n peers $PEER_POD -- bash -c "CORE_PEER_MSPCONFIGPATH=\$ADMIN_MSP_PATH peer chaincode instantiate -n mycc -o orderer.toonu.com:7050 -C mychannel -l node -v 1.0 -c '{\"Args\":[\"init\",\"a\",\"100\",\"b\",\"200\"]}' -P \"OR ('PeerMSP.member')\""
kubectl exec -n peers $PEER_POD -- bash -c "CORE_PEER_MSPCONFIGPATH=\$ADMIN_MSP_PATH peer chaincode invoke -C mychannel -n mycc -c '{"Args":["invoke","a", "b", "30"]}'"
```
gentios (Mon, 27 Jan 2020 09:06:39 GMT):
Great thank you @lionelronaldo I was thinking to install it with node sdk but this is also a good solutions tbh
lionelronaldo (Mon, 27 Jan 2020 12:48:58 GMT):
Hi! Does anyone have any tips on how to communicate with peers and orderers over an ingress?
It works great when I use a LoadBalancer, but I can't seem to configure the nginx ingress controller right so that it behaves the same as if I use the plain LoadBalancer :pray:
saanvijay (Tue, 28 Jan 2020 10:40:16 GMT):
Has joined the channel.
braduf (Fri, 07 Feb 2020 15:22:44 GMT):
Why are people here trying to mount Fabric with Kubernetes if peers communicate P2P through gossip and load is balanced by the SDK itself using the discovery service? I don't see any benefits from using Kubernetes for peers or orderers, I would only use it for de SDK services, not for the nodes. Or can someone mention why they think it is better to use Kubernetes and not just launch a machine per node, create autoscaling groups with good monitoring but without orchestration? I feel like people are just using Kubernetes because they know it and not because it is fit for Fabric....just my thoughts, it could be an interesting discussion...
braduf (Fri, 07 Feb 2020 15:22:44 GMT):
Why are people here trying to mount Fabric with Kubernetes if peers communicate P2P through gossip and load is balanced by the SDK itself using the discovery service? I don't see any benefits from using Kubernetes for peers or orderers, I would only use it for the SDK services, not for the nodes. Or can someone mention why they think it is better to use Kubernetes and not just launch a machine per node, create autoscaling groups with good monitoring but without orchestration? I feel like people are just using Kubernetes because they know it and not because it is fit for Fabric....just my thoughts, it could be an interesting discussion...
vanitas92 (Mon, 10 Feb 2020 09:50:00 GMT):
Hi everyone, i would like to setup the peers 2.0 version in kubernetes yaml the `CORE_CHAINCODE_EXTERNALBUILDERS` through env vars but i am contslanty getting the following error `Error: '': source data must be an array or slice, got string`. I put the following input:
```
- name: CORE_CHAINCODE_EXTERNALBUILDERS
value: '[{name: golang-builder, path: /builders/golang}]'
```
I think that is because env var do not support arrays but i am not completely sure. The docs suggest to modify the `core.yaml` file itself but that does not have a good approach with kubernetes as it is the only config option to be modified in `core.yaml` file itself, though i have tried that and it works:
> Modify the chaincode stanza of the peer core.yaml file to include the externalBuilders configuration element:
```
externalBuilders:
- name: myexternal
path:
rpocase (Mon, 10 Feb 2020 18:16:16 GMT):
Has any one had any luck hooking up HLF in k8s with istio? I'm trying to add just observability support (not mTLS yet), but once I have sidecar injection in place I can't seem to get traffic to my orderer
rpocase (Mon, 10 Feb 2020 18:31:06 GMT):
@iramiller maybe? Found a discussion back from May. I don't think anything has really changed in the landscape unless we were running HLF 2.0 (not there yet, the migration path is pretty steep). I really expected this to be pretty seamless if mTLS isn't deployed, but the best I can tell is the sidecar is preventing traffic from reaching the orderer
ownspies (Tue, 11 Feb 2020 17:58:56 GMT):
Has joined the channel.
ownspies (Tue, 11 Feb 2020 18:13:30 GMT):
@braduf we picked Kubernetes, at least for preproduction, to avoid having to setup a large number of EC2 instances (we're in AWS) - we can run the workload on 5 servers instead of the 27 (three orderers, six orgs, two peers per org, two CAs per org) we need if we do standalone EC2. We also selected K8S because it makes the platform slightly more cloud agnostic (we're planning to eventually leverage AWS, Azure and GCP) - no need to reinvent the wheel for each cloud provider - just have to reinvent the K8S deploy part. And while you can do Autoscale groups, you essentially either have to work out some complex cloud init scripts to auto-detect or do one ASG per system which I've done before but is a bit tedious, IMO.
ownspies (Tue, 11 Feb 2020 18:17:34 GMT):
When running peer commands, even on the local peer, we're seeing occasional `Error: error getting endorser client for list: endorser client failed to connect to peer-01.org.env.domain.com:7051: failed to create new connection: context deadline exceeded` messages for our setup. We're using HLF 1.4.4 on Kubernetes; each deployed node (orderer, peer) is fronted by an ELB in TCP pass through mode so we can use mTLS. Has anyone else run into this problem? It happens frequently enough to be concerned but not enough that the system fails completely. Note that the hostname points to the LB, so any peer commands run on the peer itself will reach out to the ELB -> K8S Service -> K8S POD.
braduf (Tue, 11 Feb 2020 18:48:46 GMT):
But so everyone who is using K8S is not having a decentralized network from what I hear. All orgs are managed by the same "operator", I think this should not be the standard way Fabric is promoted... I think it is better to start building opensource IaC for different cloud environments to make it easier for every organization to deploy it's own peers and orderer without Kubernetes. Since Fabric is supposed to be a descentralized peer-2-peer network and Kubernetes is not build for peer-2-peer...
braduf (Tue, 11 Feb 2020 18:50:06 GMT):
I think with good IaC, you also eliminate the manual setup of EC2 instances.
ownspies (Tue, 11 Feb 2020 18:55:33 GMT):
Yes with IaC but that still is more to manage, been there, done that and it only simplifies a bit
ownspies (Tue, 11 Feb 2020 18:56:11 GMT):
our K8S setup is separate namespaces per org when we run the orgs but separate K8S clusters in different AWS / Azure / On Prem accounts when the org is an external party
ownspies (Tue, 11 Feb 2020 18:56:32 GMT):
for preprod, we manage it all, for prod, we don't
ownspies (Tue, 11 Feb 2020 18:56:32 GMT):
for preprod, we manage it all, for prod, we won't
woodyjon (Tue, 11 Feb 2020 19:13:08 GMT):
Has joined the channel.
woodyjon (Tue, 11 Feb 2020 19:13:09 GMT):
Hello. Are you aware of some article, tuto or chart that implements a simple Fabric network v2.0 raft orderer on Kubernetes? I am trying to run the fabric-sample v2.0 test-network on kubernetes, but have no success.
woodyjon (Tue, 11 Feb 2020 19:13:40 GMT):
I am running into that issue: https://stackoverflow.com/questions/60172883/fabric-v2-0-in-kubernetes-minikube-error-peer-channel-error-validating-pro
ownspies (Tue, 11 Feb 2020 19:16:18 GMT):
Try running `peer channel join ...` multiple times, let me know if that works. I had a similar issue back in December and found that if I ran `peer channel join` multiple times, it worked
ownspies (Tue, 11 Feb 2020 19:16:37 GMT):
if that does indeed work for you, we should try to get a simple problem description together.
woodyjon (Tue, 11 Feb 2020 19:17:37 GMT):
No, unfortunately, I think I have tried it hundred of times already :grin:
ownspies (Tue, 11 Feb 2020 19:17:46 GMT):
ok, bummer
woodyjon (Tue, 11 Feb 2020 19:18:08 GMT):
I put all my config here: https://github.com/woodyjon/fabric-test-network-k8s
woodyjon (Tue, 11 Feb 2020 19:18:24 GMT):
I put all my config here: https://github.com/woodyjon/fabric-test-network-k8s
woodyjon (Tue, 11 Feb 2020 19:18:39 GMT):
I would be happy to make it prettier once it works
ownspies (Tue, 11 Feb 2020 19:33:04 GMT):
What do your logs on the orderer show?
woodyjon (Tue, 11 Feb 2020 19:34:15 GMT):
let me check. give me a minute
woodyjon (Tue, 11 Feb 2020 19:41:04 GMT):
the logs on the orderer does not say anything when I do the join channel for the peer0-org1
ownspies (Tue, 11 Feb 2020 19:41:30 GMT):
is FABRIC_LOGGING_SPEC=debug set on the orderer?
woodyjon (Tue, 11 Feb 2020 19:43:22 GMT):
yes
woodyjon (Tue, 11 Feb 2020 19:43:42 GMT):
"DEBUG" (is it the same as "debug")?
ownspies (Tue, 11 Feb 2020 19:43:52 GMT):
yeah
woodyjon (Tue, 11 Feb 2020 19:44:40 GMT):
it gives me that kind of things:
active nodes in cluster are: [1] channel=system-channel node=1
[36m2020-02-11 20:44:01.425 CET [orderer.consensus.etcdraft] Check -> DEBU 7c0[0m Current active nodes in cluster are: [1] channel=mychannel node=1
[36m2020-02-11 20:44:01.804 CET [orderer.consensus.etcdraft] Check -> DEBU 7c1[0m Current active nodes in cluster are: [1] channel=system-channel node=1
[36m2020-02-11 20:44:03.425 CET [orderer.consensus.etcdraft] Check -> DEBU 7c2[0m Current active nodes in cluster are: [1] channel=mychannel node=1
[36m2020-02-11 20:44:03.804 CET [orderer.consensus.etcdraft] Check -> DEBU 7c3[0m Current active nodes in cluster are: [1] channel=system-channel node=1
[36m2020-02-11 20:44:05.425 CET [orderer.consensus.etcdraft] Check -> DEBU 7c4[0m Current active nodes in cluster are: [1] channel=mychannel node=1
[36m2020-02-11 20:44:05.803 CET [orderer.consensus.etcdraft] Check -> DEBU 7c5[0m Current active nodes in cluster are: [1] channel=system-channel node=1
[36m2020-02-11 20:44:07.425 CET [orderer.consensus.etcdraft] Check -> DEBU 7c6[0m Current active nodes in cluster are: [1] channel=mychannel node=1
[36m2020-02-11 20:44:07.804 CET [orderer.consensus.etcdraft] Check -> DEBU 7c7[0m Current active nodes in cluster are: [1] channel=system-channel node=1
[36m2020-02-11 20:44:09.426 CET [orderer.consensus.etcdraft] Check -> DEBU 7c8[0m Current active nodes in cluster are: [1] channel=mychannel node=1
[36m2020-02-11 20:44:09.804 CET [orderer.consensus.etcdraft] Check -> DEBU 7c9[0m Current active nodes in cluster are: [1] channel=system-channel node=1
[36m2020-02-11 20:44:11.425 CET [orderer.consensus.etcdraft] Check -> DEBU 7ca[0m Current active nodes in cluster are: [1] channel=mychannel node=1
[36m2020-02-11 20:44:11.804 CET [orderer.consensus.etcdraft] Check -> DEBU 7cb[0m Current active nodes in cluster are: [1] channel=system-channel node=1
[36m2020-02-11 20:44:13.425 CET [orderer.consensus.etcdraft] Check -> DEBU 7cc[0m Current active nodes in cluster are: [1] channel=mychannel node=1
woodyjon (Tue, 11 Feb 2020 19:45:34 GMT):
to be clear, I am on the fabric-tools pod, and I call "peer channel join" for the peer0org1
ownspies (Tue, 11 Feb 2020 19:46:25 GMT):
did any peers successfully join the channel ?
woodyjon (Tue, 11 Feb 2020 19:48:04 GMT):
no
woodyjon (Tue, 11 Feb 2020 19:48:11 GMT):
channel creation was ok
woodyjon (Tue, 11 Feb 2020 19:48:15 GMT):
but not joining
woodyjon (Tue, 11 Feb 2020 19:48:26 GMT):
and I cannot list the channels either, thing
ownspies (Tue, 11 Feb 2020 19:48:52 GMT):
channel creation happens on the orderers effectively, IIRC
woodyjon (Tue, 11 Feb 2020 19:49:01 GMT):
yes
woodyjon (Tue, 11 Feb 2020 19:49:32 GMT):
it's a certificate problem with the peers, I think
ownspies (Tue, 11 Feb 2020 19:49:55 GMT):
do you have a cert file in `$DIR_CRYPTO_MATERIAL/peerOrganizations/org1-example-com/users/Admin@org1-example-com/msp/admincerts` ?
ownspies (Tue, 11 Feb 2020 19:50:04 GMT):
it should be the same cert that is in `$DIR_CRYPTO_MATERIAL/peerOrganizations/org1-example-com/users/Admin@org1-example-com/msp/signcerts`
woodyjon (Tue, 11 Feb 2020 19:52:15 GMT):
no, nothing there indeed
woodyjon (Tue, 11 Feb 2020 19:53:21 GMT):
in /signcerts, I have the following file: Admin@org1-example-com-cert.pem
ownspies (Tue, 11 Feb 2020 19:53:38 GMT):
copy that to admincerts
woodyjon (Tue, 11 Feb 2020 19:56:38 GMT):
I just did it. But same problem
woodyjon (Tue, 11 Feb 2020 19:57:11 GMT):
and the logs on the peer are the same
woodyjon (Tue, 11 Feb 2020 19:57:27 GMT):
access denied: channel the supplied identity is not valid: x509: certificate signed by unknown authority channel= txID=3f1376c9
[34m2020-02-11 20:56:44.937 CET [comm.grpc.server] 1 -> INFO 919[0m unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=172.17.0.6:59992 error="error validating proposal: access denied: channel [] creator org [Org1MSP]" grpc.code=Unknown grpc.call_duration=827.891µs
ownspies (Tue, 11 Feb 2020 19:58:01 GMT):
I'm looking at my scripts ... I worked through this like 8 months ago so it's a bit fuzzy
woodyjon (Tue, 11 Feb 2020 19:58:11 GMT):
thanks a lot
ownspies (Tue, 11 Feb 2020 19:58:31 GMT):
I put it all in scripts and don't have to think about it anymore, like what you're doing... bad news is I forget. But I had the exact same issue
woodyjon (Tue, 11 Feb 2020 20:03:07 GMT):
the thing is that I find a lot of things about old versions of Fabric, but as I am starting a new project, I would like to do it with 2.0. And there, I don't find much. But maybe my issue is not at all related to the version.
ownspies (Tue, 11 Feb 2020 20:03:34 GMT):
this issue is old / common
ownspies (Tue, 11 Feb 2020 20:03:46 GMT):
you're missing a CA file somewhere
woodyjon (Tue, 11 Feb 2020 20:06:14 GMT):
ok
ownspies (Tue, 11 Feb 2020 20:09:05 GMT):
You don't have CORE_PEER_MSPCONFIGPATH set inside the peer nodes
ownspies (Tue, 11 Feb 2020 20:09:17 GMT):
looking at https://github.com/woodyjon/fabric-test-network-k8s/blob/master/k8s/peer0-org1-deploy.yaml
woodyjon (Tue, 11 Feb 2020 20:10:47 GMT):
no indeed. I have FABRIC_CFG_PATH
woodyjon (Tue, 11 Feb 2020 20:10:47 GMT):
no indeed.
woodyjon (Tue, 11 Feb 2020 20:12:02 GMT):
but when I try the "peer channel join" directly when I am in the peer node, I export it
woodyjon (Tue, 11 Feb 2020 20:12:32 GMT):
I will retry with the env variable
ownspies (Tue, 11 Feb 2020 20:12:56 GMT):
so there are two things I *think* I dealt with when I had this problem and I can't recall which relates to the endorser
woodyjon (Tue, 11 Feb 2020 20:13:28 GMT):
can I just delete the peer deployment and redeploy it with the new config or do I need to destroy everything and restart?
ownspies (Tue, 11 Feb 2020 20:14:00 GMT):
1) the admin cert sent from the tools container is signed by a CA that the peer doesn't recognize (I'd have to sort through all the configs you have to see what you're using where)
2) the cert sent by the peer to the orderer is signed by a CA that the orderer does not recognize
ownspies (Tue, 11 Feb 2020 20:14:22 GMT):
#2 seems more likely given that typically the Admin cert is signed by the same CA that signed the peer certs
ownspies (Tue, 11 Feb 2020 20:14:42 GMT):
those CAs are all embedded in the genesis block and then again in the channel creation blocks
woodyjon (Tue, 11 Feb 2020 20:16:10 GMT):
but I used cryptogen to generate the ca stuff
ownspies (Tue, 11 Feb 2020 20:16:33 GMT):
yeah, I see you used separate files; I always used a single file, but I would think it doesn't matter
woodyjon (Tue, 11 Feb 2020 20:16:46 GMT):
and would it be possible that the peer does not have read access to the certificates ? Everything is on a common nfs server
woodyjon (Tue, 11 Feb 2020 20:17:13 GMT):
those are the same files as the test-network of the fabric-sample folder
woodyjon (Tue, 11 Feb 2020 20:17:30 GMT):
the only thing I changed in those files is change the dots to dashes
ownspies (Tue, 11 Feb 2020 20:17:47 GMT):
when I had this problem, I also went through *all* of my certs and ran `openssl x509 -noout -text -in` and validated *everything* matched :(
ownspies (Tue, 11 Feb 2020 20:17:51 GMT):
it was tedious
woodyjon (Tue, 11 Feb 2020 20:17:54 GMT):
peer0.org1.example.com to peer0-org1-example-com
woodyjon (Tue, 11 Feb 2020 20:18:05 GMT):
ok I could do that
woodyjon (Tue, 11 Feb 2020 20:18:12 GMT):
how do you check that it matches?
ownspies (Tue, 11 Feb 2020 20:19:00 GMT):
the short way is to validate timestamps are the same / near (off by 1 second or 2) and make sure the CA names in the issuer field matches the name on the CA cert
ownspies (Tue, 11 Feb 2020 20:19:08 GMT):
the long way is to calculate MD5 sums
ownspies (Tue, 11 Feb 2020 20:20:31 GMT):
I do see that https://github.com/woodyjon/fabric-test-network-k8s/blob/master/k8s/peer0-org1-deploy.yaml has `CORE_PEER_TLS_ENABLED` set to true, but https://github.com/woodyjon/fabric-test-network-k8s/blob/master/k8s/fabric-tools.yaml has it set to false, do you manually set that when you use the CLI?
woodyjon (Tue, 11 Feb 2020 20:21:53 GMT):
yes, I manually set it to true
woodyjon (Tue, 11 Feb 2020 20:22:01 GMT):
but I will change it there as well
ownspies (Tue, 11 Feb 2020 20:23:31 GMT):
ahh
ownspies (Tue, 11 Feb 2020 20:23:34 GMT):
perhaps this:
ownspies (Tue, 11 Feb 2020 20:23:34 GMT):
nevermind
woodyjon (Tue, 11 Feb 2020 20:24:09 GMT):
:pray:
woodyjon (Tue, 11 Feb 2020 20:24:12 GMT):
ho shit
woodyjon (Tue, 11 Feb 2020 20:24:14 GMT):
;-)
ownspies (Tue, 11 Feb 2020 20:24:27 GMT):
wait, was trying to double check inside this chat
ownspies (Tue, 11 Feb 2020 20:24:30 GMT):
but it's harder to read
ownspies (Tue, 11 Feb 2020 20:24:32 GMT):
```export CORE_PEER_MSPCONFIGPATH=$DIR_CRYPTO_MATERIAL/peerOrganizations/org1-example-com/users/Admin@org1-example-com/msp
#export CORE_PEER_MSPCONFIGPATH=$DIR_CRYPTO_MATERIAL/peerOrganizations/org1-example-com/peers/peer0-org1-example-com/msp/
export CORE_PEER_ADDRESS=peer0-org1-example-com:7051```
ownspies (Tue, 11 Feb 2020 20:25:27 GMT):
ok, no that's fine, I was thinking it was using org1 MSP and org0 for the peer address
ownspies (Tue, 11 Feb 2020 20:25:48 GMT):
i
ownspies (Tue, 11 Feb 2020 20:25:48 GMT):
I'm not a fan of mixing zero based and ones based numbering (e.g. first peer is peer0, first org is org1)
ownspies (Tue, 11 Feb 2020 20:26:24 GMT):
HLF started that trend, drove me nuts reading stuff
woodyjon (Tue, 11 Feb 2020 20:27:37 GMT):
yes I agree. that's also from their test-network
ownspies (Tue, 11 Feb 2020 20:27:38 GMT):
have you tried this command:
ownspies (Tue, 11 Feb 2020 20:28:01 GMT):
```peer channel join -b $CHANNEL_NAME.block -o orderer-example-com:7050 -c $CHANNEL_NAME --tls $CORE_PEER_TLS_ENABLED --cafile $ORDERER_CA```
ownspies (Tue, 11 Feb 2020 20:29:08 GMT):
usually the channel create and channel join should be the same basic commands
ownspies (Tue, 11 Feb 2020 20:29:11 GMT):
I ran into that issue too
woodyjon (Tue, 11 Feb 2020 20:29:56 GMT):
yes I did that too
ownspies (Tue, 11 Feb 2020 20:30:37 GMT):
and you get the same error ?
woodyjon (Tue, 11 Feb 2020 20:31:41 GMT):
yes
woodyjon (Tue, 11 Feb 2020 20:32:24 GMT):
would it be possible that it is an issue of the peer trying to connect to himself for endorsement?
ownspies (Tue, 11 Feb 2020 20:35:02 GMT):
I don't believe so, even if that were the case the peer should already be talking to itself
ownspies (Tue, 11 Feb 2020 20:35:08 GMT):
since you installed all the certs
woodyjon (Tue, 11 Feb 2020 20:36:20 GMT):
the certs, they are just put by the fabric-tools pod on a shared nfs server
ownspies (Tue, 11 Feb 2020 20:36:34 GMT):
yeah
woodyjon (Tue, 11 Feb 2020 20:38:49 GMT):
I'm very grateful that you take the time to look at this. Big thanks!
ownspies (Tue, 11 Feb 2020 20:39:31 GMT):
sure, it's hard to troubleshoot over chat ... HLF tends to output important information long before the actual error
ownspies (Tue, 11 Feb 2020 20:39:47 GMT):
you almost need to post the entire output when you run the command with debug mode
ownspies (Tue, 11 Feb 2020 20:39:52 GMT):
but ... that is a lot of info
ownspies (Tue, 11 Feb 2020 20:40:07 GMT):
I've solved many problems by scrolling up 20, 30, 50 lines
ownspies (Tue, 11 Feb 2020 20:40:32 GMT):
can you run this and paste the output:
ownspies (Tue, 11 Feb 2020 20:40:34 GMT):
```peer channel join -b $CHANNEL_NAME.block -o orderer-example-com:7050 -c $CHANNEL_NAME --tls $CORE_PEER_TLS_ENABLED --cafile $ORDERER_CA```
ownspies (Tue, 11 Feb 2020 20:41:29 GMT):
this is what I use to create the channel (I have mTLS enabled)
ownspies (Tue, 11 Feb 2020 20:41:35 GMT):
```
export WORKENV=dev
export CORE_PEER_ADDRESS="peer-01.$ORG.dev.example.net:7051"
export CORE_PEER_MSPCONFIGPATH="/hyperledger/crypto-config/peerOrganizations/$ORG.dev.example.net/msp"
export CORE_PEER_NETWORKID="nid1"
if [ -z $FABRIC_LOGGING_SPEC ] ; then
export FABRIC_LOGGING_SPEC="INFO"
fi
export CORE_PEER_TLS_ENABLED="true"
export CORE_PEER_TLS_CLIENTROOTCAS_FILES="[/hyperledger/crypto-config/peerOrganizations/$ORG.dev.example.net/tlsca/tlsca.$ORG.dev.example.net-cert.pem]"
export CORE_PEER_TLS_CLIENTCERT_FILE="/hyperledger/crypto-config/peerOrganizations/$ORG.dev.example.net/users/admin/tls/client.crt"
export CORE_PEER_TLS_CLIENTKEY_FILE="/hyperledger/crypto-config/peerOrganizations/$ORG.dev.example.net/users/admin/tls/client.key"
export CORE_PEER_TLS_ROOTCERT_FILE="/hyperledger/crypto-config/peerOrganizations/$ORG.dev.example.net/tlsca/tlsca.$ORG.dev.example.net-cert.pem"
printf '==================================================\n Creating Channel %s\n==================================================\n' "$CHANNEL_NAME"
peer channel create \
--outputBlock /var/hyperledger/${CHANNEL_NAME}.block \
-c $CHANNEL_NAME \
-f /hyperledger/channel-artifacts/channel-${CHANNEL_NAME}.tx \
-o orderer-01.${WORKENV}.example.net:7050 \
--tls --cafile /hyperledger/crypto-config/ordererOrganizations/dev.example.net/tlsca/tlsca.dev.example.net-cert.pem \
--clientauth \
--keyfile $CORE_PEER_TLS_CLIENTKEY_FILE \
--certfile $CORE_PEER_TLS_CLIENTCERT_FILE
```
woodyjon (Tue, 11 Feb 2020 20:41:50 GMT):
2020-02-11 21:41:31.161 CET [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
Error: proposal failed (err: rpc error: code = Unknown desc = error validating proposal: access denied: channel [] creator org [Org1MSP])
woodyjon (Tue, 11 Feb 2020 20:42:01 GMT):
and on the peer node:
woodyjon (Tue, 11 Feb 2020 20:42:27 GMT):
18012A500A3F0A1B70656572302D6F72...53120D08C9B3AFEFAA8B9CF91510A00D
[36m2020-02-11 21:42:00.656 CET [msp.identity] Sign -> DEBU 1ef0[0m Sign: digest: E206F0778A2923A3E48AC8A67A548610B34DF0142D5E3A2D7A1F8FC5552AF8EC
[36m2020-02-11 21:42:00.657 CET [msp.identity] Sign -> DEBU 1ef1[0m Sign: plaintext: 0A1B70656572302D6F7267312D6578616D706C652D636F6D3A37303531
[36m2020-02-11 21:42:00.657 CET [msp.identity] Sign -> DEBU 1ef2[0m Sign: digest: 720F5387E762A50BDE1E42F9839673B7DE61CF7993E0F2F0AFBED4522F3BD14B
[36m2020-02-11 21:42:00.657 CET [gossip.discovery] periodicalSendAlive -> DEBU 1ef3[0m Sleeping 5s
[36m2020-02-11 21:42:03.511 CET [endorser] ProcessProposal -> DEBU 1ef4[0m request from 172.17.0.6:49640
[36m2020-02-11 21:42:03.511 CET [msp] DeserializeIdentity -> DEBU 1ef5[0m Obtaining identity
[36m2020-02-11 21:42:03.511 CET [msp.identity] newIdentity -> DEBU 1ef6[0m Creating identity instance for cert -----BEGIN CERTIFICATE-----
MIICKTCCAdCgAwIBAgIRAOA1bGZ4QaK8gJskYwxiF6cwCgYIKoZIzj0EAwIwczEL
MAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcTDVNhbiBG
cmFuY2lzY28xGTAXBgNVBAoTEG9yZzEtZXhhbXBsZS1jb20xHDAaBgNVBAMTE2Nh
Lm9yZzEtZXhhbXBsZS1jb20wHhcNMjAwMjExMTkzNDAwWhcNMzAwMjA4MTkzNDAw
WjBrMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTEWMBQGA1UEBxMN
U2FuIEZyYW5jaXNjbzEOMAwGA1UECxMFYWRtaW4xHzAdBgNVBAMMFkFkbWluQG9y
ZzEtZXhhbXBsZS1jb20wWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAAQ/0aZM42x8
EX3JwH7lT5a6R2EygV9rtNJb5Gg4pvr9+yiaPFGGrhXlCmAXf/WzVHFJGcpaCU19
n3AI9JO13BuBo00wSzAOBgNVHQ8BAf8EBAMCB4AwDAYDVR0TAQH/BAIwADArBgNV
HSMEJDAigCB5M0OGrHT9TGUNlQICx+9hC2sLdCi1z/zzoW8yiTwlozAKBggqhkjO
PQQDAgNHADBEAiB1+yZu5USM+VhJVSnKkVKzuen1XDrmfJB+NZVqz+il9AIgckE5
7wKkXtmvUqrK4it9Zqu/Sadza/T6p2oBvN7eW7k=
-----END CERTIFICATE-----
[33m2020-02-11 21:42:03.511 CET [endorser] Validate -> WARN 1ef7[0m access denied: channel the supplied identity is not valid: x509: certificate signed by unknown authority channel= txID=6fb502d4
[34m2020-02-11 21:42:03.511 CET [comm.grpc.server] 1 -> INFO 1ef8[0m unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=172.17.0.6:49640 error="error validating proposal: access denied: channel [] creator org [Org1MSP]" grpc.code=Unknown grpc.call_duration=643.916µs
[36m2020-02-11 21:42:03.513 CET [grpc] infof -> DEBU 1ef9[0m transport: loopyWriter.run returning. connection error: desc = "transport is closing"
[36m2020-02-11 21:42:03.894 CET [msp.identity] Sign -> DEBU 1efa[0m Sign: plaintext: 18012A500A3F0A1B70656572302D6F72...53120D08C9B3AFEFAA8B9CF91510A10D
[36m2020-02-11 21:42:03.894 CET [msp.identity] Sign -> DEBU 1efb[0m Sign: digest: 24A393B83CA39557706D98E19D7CD1AA8E21922643CE33C22AC18285916DD01E
[36m2020-02-11 21:42:03.895 CET [msp.identity] Sign -> DEBU 1efc[0m Sign: plaintext: 0A1B70656572302D6F7267312D6578616D706C652D636F6D3A37303531
[36m2020-02-11 21:42:03.895 CET [msp.identity] Sign -> DEBU 1efd[0m Sign: digest: 720F5387E762A50BDE1E42F9839673B7DE61CF7993E0F2F0AFBED4522F3BD14B
[36m2020-02-11 21:42:05.657 CET [msp.identity] Sign -> DEBU 1efe[0m Sign: plaintext: 18012A500A3F0A1B70656572302D6F72...53120D08C9B3AFEFAA8B9CF91510A20D
[36m2020-02-11 21:42:05.657 CET [msp.identity] Sign -> DEBU 1eff[0m Sign: digest: 299A8E95F44F43D41E740F49D901622D2D31DA65B0494A47E19877316941666F
[36m2020-02-11 21:42:05.657 CET [msp.identity] Sign -> DEBU 1f00[0m Sign: plaintext: 0A1B70656572302D6F7267312D6578616D706C652D636F6D3A37303531
[36m2020-02-11 21:42:05.657 CET [msp.identity] Sign -> DEBU 1f01[0m Sign: digest: 720F5387E762A50BDE1E42F9839673B7DE61CF7993E0F2F0AFBED4522F3BD14B
[36m2020-02-11 21:42:05.657 CET [gossip.discovery] periodicalSendAlive -> DEBU 1f02[0m Sleeping 5s
[36m2020-02-11 21:42:07.896 CET [msp.identity] Sign -> DEBU 1f03[0m Sign: plaintext: 18012A500A3F0A1B70656572302D6F72...53120D08C9B3AFEFAA8B9CF91510A30D
[36m2020-02-11 21:42:07.896 CET [msp.identity] Sign -> DEBU 1f04[0m Sign: digest: 95E52EDA084305E6B467A136D18BB9B4C6F1DF4B7D7D56E908DDC03DC63E7513
[36m2020-02-11 21:42:07.896 CET [msp.identity] Sign -> DEBU 1f05[0m Sign: plaintext: 0A1B70656572302D6F7267312D6578616D706C652D636F6D3A37303531
[36m2020-02-11 21:42:07.896 CET [msp.identity] Sign -> DEBU 1f06[0m Sign: digest: 720F5387E762A50BDE1E42F9839673B7DE61CF7993E0F2F0AFBED4522F3BD14B
[36m2020-02-11 21:42:09.054 CET [gossip.discovery] periodicalReconnectToDead -> DEBU 1f07[0m Sleeping 25s
woodyjon (Tue, 11 Feb 2020 20:44:56 GMT):
do you know the need for export CORE_PEER_NETWORKID="nid1"?
ownspies (Tue, 11 Feb 2020 20:45:22 GMT):
No, I haven't had to change that so I haven't read up on it
woodyjon (Tue, 11 Feb 2020 20:45:39 GMT):
I don't do CORE_PEER_TLS_CLIENTROOTCAS_FILES
woodyjon (Tue, 11 Feb 2020 20:45:42 GMT):
do I need it?
ownspies (Tue, 11 Feb 2020 20:45:52 GMT):
no because you're not using mTLS
woodyjon (Tue, 11 Feb 2020 20:46:29 GMT):
client TLS, you mean?
woodyjon (Tue, 11 Feb 2020 20:47:10 GMT):
ha ok, mutual TLS, so client as well
ownspies (Tue, 11 Feb 2020 20:47:13 GMT):
yes / Mutual TLS
woodyjon (Tue, 11 Feb 2020 20:48:10 GMT):
your setup is also on kubernetes?
ownspies (Tue, 11 Feb 2020 20:48:23 GMT):
yes, but a full cluster and I use the helm charts
ownspies (Tue, 11 Feb 2020 20:48:32 GMT):
well, highly customized versions of them
ownspies (Tue, 11 Feb 2020 20:49:03 GMT):
and I run the peer commands on a peer node in k8s or in a local docker container with all the ENV vars set
woodyjon (Tue, 11 Feb 2020 20:49:07 GMT):
I did not manage to get deployments with dots in the names, that's why i changed every names and addresses to dashes
ownspies (Tue, 11 Feb 2020 20:49:11 GMT):
you should try running the join command on the peer itself
ownspies (Tue, 11 Feb 2020 20:49:23 GMT):
if that doesn't work, then it probably is that the peer can't talk to the orderer
woodyjon (Tue, 11 Feb 2020 20:49:39 GMT):
I tried on the peer itself, and same error
ownspies (Tue, 11 Feb 2020 20:50:13 GMT):
doesn't make sense that the create works but not join ...
woodyjon (Tue, 11 Feb 2020 20:50:38 GMT):
PQQDAgNHADBEAiB1+yZu5USM+VhJVSnKkVKzuen1XDrmfJB+NZVqz+il9AIgckE5
7wKkXtmvUqrK4it9Zqu/Sadza/T6p2oBvN7eW7k=
-----END CERTIFICATE-----
2020-02-11 21:50:17.518 CET [msp] hasOURole -> DEBU 01d MSP Org1MSP checking if the identity is a client
2020-02-11 21:50:17.518 CET [msp] getCertificationChain -> DEBU 01e MSP Org1MSP getting certification chain
2020-02-11 21:50:17.519 CET [msp] hasOURole -> DEBU 01f MSP Org1MSP checking if the identity is a client
2020-02-11 21:50:17.519 CET [msp] getCertificationChain -> DEBU 020 MSP Org1MSP getting certification chain
2020-02-11 21:50:17.519 CET [msp] GetDefaultSigningIdentity -> DEBU 021 Obtaining default signing identity
2020-02-11 21:50:17.521 CET [grpc] WithKeepaliveParams -> DEBU 022 Adjusting keepalive ping interval to minimum period of 10s
2020-02-11 21:50:17.521 CET [grpc] DialContext -> DEBU 023 parsed scheme: ""
2020-02-11 21:50:17.521 CET [grpc] DialContext -> DEBU 024 scheme "" not registered, fallback to default scheme
2020-02-11 21:50:17.521 CET [grpc] UpdateState -> DEBU 025 ccResolverWrapper: sending update to cc: {[{peer0-org1-example-com:7051 0
woodyjon (Tue, 11 Feb 2020 20:50:38 GMT):
PQQDAgNHADBEAiB1+yZu5USM+VhJVSnKkVKzuen1XDrmfJB+NZVqz+il9AIgckE5
7wKkXtmvUqrK4it9Zqu/Sadza/T6p2oBvN7eW7k=
-----END CERTIFICATE-----
2020-02-11 21:50:17.518 CET [msp] hasOURole -> DEBU 01d MSP Org1MSP checking if the identity is a client
2020-02-11 21:50:17.518 CET [msp] getCertificationChain -> DEBU 01e MSP Org1MSP getting certification chain
2020-02-11 21:50:17.519 CET [msp] hasOURole -> DEBU 01f MSP Org1MSP checking if the identity is a client
2020-02-11 21:50:17.519 CET [msp] getCertificationChain -> DEBU 020 MSP Org1MSP getting certification chain
2020-02-11 21:50:17.519 CET [msp] GetDefaultSigningIdentity -> DEBU 021 Obtaining default signing identity
2020-02-11 21:50:17.521 CET [grpc] WithKeepaliveParams -> DEBU 022 Adjusting keepalive ping interval to minimum period of 10s
2020-02-11 21:50:17.521 CET [grpc] DialContext -> DEBU 023 parsed scheme: ""
2020-02-11 21:50:17.521 CET [grpc] DialContext -> DEBU 024 scheme "" not registered, fallback to default scheme
2020-02-11 21:50:17.521 CET [grpc] UpdateState -> DEBU 025 ccResolverWrapper: sending update to cc: {[{peer0-org1-example-com:7051 0
woodyjon (Tue, 11 Feb 2020 20:51:57 GMT):
do you know why in the error (access denied: channel [] creator org [Org1MSP]) the array seems empty, it does not give the name of the channel?
ownspies (Tue, 11 Feb 2020 20:52:21 GMT):
not sure
ownspies (Tue, 11 Feb 2020 20:52:42 GMT):
I feel like I've seen that ... but I've seen so much that it all is a blurry picture
ownspies (Tue, 11 Feb 2020 20:53:12 GMT):
the other answer I've found is to sometimes just wipe the entire thing out and start over (aka reboot)
ownspies (Tue, 11 Feb 2020 20:53:31 GMT):
I don't think it will fix the issue, but maybe it will knock an idea loose
ownspies (Tue, 11 Feb 2020 20:53:47 GMT):
I would also turn off TLS for now
ownspies (Tue, 11 Feb 2020 20:53:53 GMT):
get the most basic thing working
ownspies (Tue, 11 Feb 2020 20:53:55 GMT):
then add TLS back
ownspies (Tue, 11 Feb 2020 20:54:34 GMT):
leave the RAFT TLS on, but turn off TLS between CLI / Peer / Orderer
ownspies (Tue, 11 Feb 2020 20:55:42 GMT):
you can also decrypt the genesis block and the channel create blocks
ownspies (Tue, 11 Feb 2020 20:56:06 GMT):
that will output JSON, then you have to extract the certs from the JSON and compare them to what you have on disk
woodyjon (Tue, 11 Feb 2020 20:56:18 GMT):
ok I will try that
woodyjon (Tue, 11 Feb 2020 20:56:28 GMT):
wiping out, I've already did it many times
woodyjon (Tue, 11 Feb 2020 20:56:42 GMT):
tls cannot be turned off, I think, for raft
woodyjon (Tue, 11 Feb 2020 20:56:50 GMT):
I think I tried
woodyjon (Tue, 11 Feb 2020 20:57:13 GMT):
thanks a lot anyway!!
ownspies (Tue, 11 Feb 2020 20:57:19 GMT):
are you running > 1 orderer ?
woodyjon (Tue, 11 Feb 2020 20:57:23 GMT):
no
ownspies (Tue, 11 Feb 2020 20:57:25 GMT):
if not, turn off RAFT and TLS
woodyjon (Tue, 11 Feb 2020 20:57:40 GMT):
so use kafka?
ownspies (Tue, 11 Feb 2020 20:57:48 GMT):
yes, I was trying to remember if TLS was required for RAFT only or if it had to be on for the entire orderer
ownspies (Tue, 11 Feb 2020 20:57:59 GMT):
at any rate, I think you can leave TLS on for the orderer if you need, but turn off on the peers
ownspies (Tue, 11 Feb 2020 20:58:10 GMT):
I think you can do standalone, let me look
ownspies (Tue, 11 Feb 2020 20:58:14 GMT):
no rAFT no Kafka
woodyjon (Tue, 11 Feb 2020 20:58:19 GMT):
ok
ownspies (Tue, 11 Feb 2020 20:58:31 GMT):
```Orderer: &OrdererDefaults
# Orderer Type: The orderer implementation to start
# Available types are "solo", "kafka", and "etcdraft"
OrdererType: etcdraft
```
ownspies (Tue, 11 Feb 2020 20:58:40 GMT):
set that to `solo` not `etcraft`
woodyjon (Tue, 11 Feb 2020 20:58:45 GMT):
ok I will try solo then
woodyjon (Tue, 11 Feb 2020 20:58:48 GMT):
thank you
ownspies (Tue, 11 Feb 2020 20:58:59 GMT):
pull to bare minimum, then add one or two things at a time
woodyjon (Tue, 11 Feb 2020 20:59:08 GMT):
yes, exactly
ownspies (Tue, 11 Feb 2020 20:59:33 GMT):
when you wipe it out, do you delete all the .block files and all the TLS files ?
woodyjon (Tue, 11 Feb 2020 20:59:37 GMT):
but the pb does not seem to be tls here, right?
woodyjon (Tue, 11 Feb 2020 20:59:45 GMT):
yes, I delete everything
woodyjon (Tue, 11 Feb 2020 20:59:52 GMT):
I delete minikube
ownspies (Tue, 11 Feb 2020 20:59:58 GMT):
I don't *think* it is tls, but I would disable and confirm
woodyjon (Tue, 11 Feb 2020 21:00:08 GMT):
yes ok
ownspies (Tue, 11 Feb 2020 21:00:23 GMT):
when you delete minikube, does it wipe out the contents of the NFS volume?
woodyjon (Tue, 11 Feb 2020 21:00:33 GMT):
I delete it manually
ownspies (Tue, 11 Feb 2020 21:00:36 GMT):
ok
ownspies (Tue, 11 Feb 2020 21:00:46 GMT):
try those things, see what you find out
woodyjon (Tue, 11 Feb 2020 21:00:54 GMT):
yes I've done it many times. I change small stuff then I retry
woodyjon (Tue, 11 Feb 2020 21:00:58 GMT):
yes
woodyjon (Tue, 11 Feb 2020 21:01:10 GMT):
if you have an illumination, contact me please
woodyjon (Tue, 11 Feb 2020 21:01:22 GMT):
:grinning:
ownspies (Tue, 11 Feb 2020 21:02:33 GMT):
have you done this to inspect and compare the certs?
ownspies (Tue, 11 Feb 2020 21:03:02 GMT):
```root@26350254d568:/hyperledger/channel-artifacts# configtxgen -inspectBlock genesis.block | head -20
2020-02-11 21:02:00.898 UTC [common.tools.configtxgen] main -> INFO 001 Loading configuration
2020-02-11 21:02:01.294 UTC [common.tools.configtxgen.localconfig] completeInitialization -> INFO 002 Orderer.Addresses unset, setting to [127.0.0.1:7050]
2020-02-11 21:02:01.294 UTC [common.tools.configtxgen.localconfig] completeInitialization -> INFO 003 orderer type: solo
2020-02-11 21:02:01.295 UTC [common.tools.configtxgen.localconfig] LoadTopLevel -> INFO 004 Loaded configuration: /etc/hyperledger/fabric/configtx.yaml
2020-02-11 21:02:01.295 UTC [common.tools.configtxgen] doInspectBlock -> INFO 005 Inspecting block
2020-02-11 21:02:01.298 UTC [common.tools.configtxgen] doInspectBlock -> INFO 006 Parsing genesis block
{
"data": {
"data": [
{
"payload": {
"data": {
"config": {
"channel_group": {
"groups": {
"Consortiums": {
"groups": {
"GlobalConsortium": {
"groups": {
"MyOrg": {
"groups": {},
"mod_policy": "Admins",
"policies": {
"Admins": {
"mod_policy": "Admins",
"policy": {
```
woodyjon (Tue, 11 Feb 2020 21:03:17 GMT):
no
woodyjon (Tue, 11 Feb 2020 21:05:33 GMT):
do you know what I should check in there?
woodyjon (Tue, 11 Feb 2020 21:06:01 GMT):
mmm ok, it's quite long but I'll check it
ownspies (Tue, 11 Feb 2020 21:07:07 GMT):
I compared the CA certs for the orderers and the orgs to the files on disk; it's a long shot for you, but in my case K8S seemed to be caching the TLS certs or my process (more likely) was failing to update them correctly in K8S
ownspies (Tue, 11 Feb 2020 21:07:19 GMT):
sometimes the file in K8S didn't match what I had on disk locally
woodyjon (Tue, 11 Feb 2020 21:07:42 GMT):
ok
ownspies (Tue, 11 Feb 2020 21:07:49 GMT):
(I generate a lot of the material locally and upload to K8S via secrets)
woodyjon (Tue, 11 Feb 2020 21:07:54 GMT):
but even if it does not match, what could I do?
ownspies (Tue, 11 Feb 2020 21:08:13 GMT):
if it doesn't match, then your process is wrong and you need to fix it
woodyjon (Tue, 11 Feb 2020 21:08:46 GMT):
:thumbsup:
ownspies (Tue, 11 Feb 2020 21:08:58 GMT):
for example, if you generate the crypto files, then generate the configs, then regenerate the crypto material, the system will be broken because you distributed old crypto material with the genesis block
woodyjon (Tue, 11 Feb 2020 21:09:10 GMT):
ok I see
woodyjon (Tue, 11 Feb 2020 21:09:39 GMT):
does the version of cryptogen matter?
ownspies (Tue, 11 Feb 2020 21:10:15 GMT):
more specific generate crypto -> generate genesis -> generate crypto -> launch services and in that situation, the services have old crypto material from the genesis block that doesn't match the stuff on disk, so things don't work; that happened to me too when I tried to take short cuts and selectively regenerate broken crypto files
ownspies (Tue, 11 Feb 2020 21:10:27 GMT):
the version of crypto gen should match the HLF system
woodyjon (Tue, 11 Feb 2020 21:10:37 GMT):
ok
woodyjon (Tue, 11 Feb 2020 21:11:56 GMT):
thanks!!
ownspies (Tue, 11 Feb 2020 21:12:05 GMT):
keep us posted!
woodyjon (Tue, 11 Feb 2020 21:12:13 GMT):
I will
woodyjon (Tue, 11 Feb 2020 21:12:28 GMT):
do you work for hyperledger?
ownspies (Tue, 11 Feb 2020 21:12:37 GMT):
nope
ownspies (Tue, 11 Feb 2020 21:12:51 GMT):
I work for a small company in the US
woodyjon (Tue, 11 Feb 2020 21:13:05 GMT):
k
ShrutiHK (Wed, 12 Feb 2020 05:36:13 GMT):
Has joined the channel.
ShrutiHK (Wed, 12 Feb 2020 05:48:54 GMT):
Hello everyone
I am trying to deploy fabric on kubernetes in a GKE cluster. I could not make the nfs folder mounting work on the k8s cluster nodes from different nfs VM. So, now I have deployed nfs server in a pod.
Still, the orderer and peer pods are not getting deployed and the 'kubectl describe pod' command shows the error as 'Failed mount'
ShrutiHK (Wed, 12 Feb 2020 05:49:25 GMT):
Can someone suggest what needs to done to resolve the issue?
ShrutiHK (Wed, 12 Feb 2020 05:52:57 GMT):
the detailed logs of the orderer pod show : 'Kafka version unset. Setting to 0.10.2.0'
ShrutiHK (Wed, 12 Feb 2020 05:53:08 GMT):
But, I am not using KAFKA at all
ShrutiHK (Wed, 12 Feb 2020 05:53:13 GMT):
I am using RAFT
woodyjon (Wed, 12 Feb 2020 08:52:53 GMT):
hello @ownspies . I successfully joined the channel!!
woodyjon (Wed, 12 Feb 2020 08:53:47 GMT):
the only thing that was missing is the CORE_PEER_MSPCONFIGPATH in the yaml files of the peers
woodyjon (Wed, 12 Feb 2020 08:53:53 GMT):
as you mentioned it yesterday!!
woodyjon (Wed, 12 Feb 2020 08:54:12 GMT):
so thanks a lot for this!
usi (Wed, 12 Feb 2020 15:19:56 GMT):
Has joined the channel.
woodyjon (Thu, 13 Feb 2020 11:48:11 GMT):
Hello all. I am still trying to deploy the fabric-sample v2.0 test-network on kubernetes. I progressed a lot, but trying to invoke the chaincode, I am facing an issue. I think it is related to the docker container that is supposed to run the chaincode. I have explained this in more details here: https://stackoverflow.com/questions/60207056/fabric-v2-0-in-kubernetes-minikube-problem-running-docker-inside-peer-for-ru . Do you have an idea? Thanks
woodyjon (Thu, 13 Feb 2020 12:50:46 GMT):
Hello all. I am still trying to deploy the fabric-sample v2.0 test-network on kubernetes. I progressed a lot, but trying to invoke the chain
indirajith (Fri, 14 Feb 2020 17:28:29 GMT):
Has joined the channel.
AbhijeetSamanta (Sat, 15 Feb 2020 14:53:52 GMT):
Hi All, I am trying to implement HLF on multicluster k8s, however anyone have experience to implement it, I have some query regarding the setup
futurama92 (Mon, 17 Feb 2020 20:29:15 GMT):
Has left the channel.
JayJong (Tue, 18 Feb 2020 07:10:57 GMT):
Hi all, i have setup fabric with kubernetes and wish to ask about high availability. When my peer pod goes down in the cluster, kubernetes will bring the peer pod up again but i have to manually do a command from the CLI to join channel and install chaincode on the peer again, is there any way i can do it automatically after it is down and brought back up?
woodyjon (Tue, 18 Feb 2020 08:41:57 GMT):
mmm good question. I also would like to hear from people used to it how do they achieve that.
vanitas92 (Tue, 18 Feb 2020 10:22:35 GMT):
this question has been asked and answered many times here, please read the following entry in the official documentation: https://hyperledger-fabric.readthedocs.io/en/release-1.4/build_network.html#a-note-on-data-persistence
woodyjon (Tue, 18 Feb 2020 10:32:31 GMT):
thanks @vanitas92 !
JayJong (Tue, 18 Feb 2020 10:34:14 GMT):
@vanitas92 i think u may have been mistaken, im not asking how the data can be persisted but rather some way for the peer pod to join back the channel instead of me typing into the CLI pod
vanitas92 (Tue, 18 Feb 2020 10:35:32 GMT):
if you persist that data, when the pod restarts you wont need to do that manually when u back up the data
vanitas92 (Tue, 18 Feb 2020 10:35:32 GMT):
if you persist that data, when the pod restarts you wont need to do that manually when u persist the data
woodyjon (Tue, 18 Feb 2020 18:12:28 GMT):
hello @vanitas92 I checked what they say here https://hyperledger-fabric.readthedocs.io/en/release-1.4/build_network.html#a-note-on-data-persistence but I don't understand how to apply this on kubernetes. Could you explain how to put that in practice for kubernetes?
vanitas92 (Wed, 19 Feb 2020 09:18:59 GMT):
so you need to mount a volumemount inside the pod and then map it to the backing storage that you are using, see an example of that by using hostpath:
```
volumeMounts:
- mountPath: /var/hyperledger/production
name: peer0-persistentdata
volumes:
- name: peer0-persistentdata
hostPath:
path: /yourPath
type: DirectoryOrCreate
```
vanitas92 (Wed, 19 Feb 2020 09:18:59 GMT):
so you need to mount a volumemount inside the pod and then map it to the backing storage that you are using, see an example of that by using hostpath:
```yaml
volumeMounts:
- mountPath: /var/hyperledger/production
name: peer0-persistentdata
volumes:
- name: peer0-persistentdata
hostPath:
path: /yourPath
type: DirectoryOrCreate
```
woodyjon (Wed, 19 Feb 2020 10:20:09 GMT):
thank you
JayJong (Thu, 20 Feb 2020 10:54:14 GMT):
@vanitas92 i have tested and verified wad u say is correct, thank you for your help!
woodyjon (Thu, 20 Feb 2020 10:58:30 GMT):
isn't it easier to just juse statefulsets instead of deployments, so that the persistence aspect is built in?
vanitas92 (Thu, 20 Feb 2020 11:03:00 GMT):
Statefulset is just a way on handling k8s pods in case of restarting or worker faults, it is recommended for stateful applications rather than stateless apps. Peers are stateful in that sense. However, statefulsets DO NOT guarantee persistence by themselves alone, you have to backup the data somehow using the type of storage your application might need.
vanitas92 (Thu, 20 Feb 2020 11:03:43 GMT):
That could be NFS, block storage or whatever your application might need
vanitas92 (Thu, 20 Feb 2020 11:05:04 GMT):
you can read more on that here: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/
woodyjon (Thu, 20 Feb 2020 13:53:19 GMT):
:thumbsup:
ownspies (Fri, 21 Feb 2020 21:10:26 GMT):
I'm interested to know - for anyone using K8S, are you using external LBs to access the HLF nodes (service type LoadBalancer)? If so, which LB are you using? Thinking I may need to move off AWS NLB... :/
lionelronaldo (Mon, 24 Feb 2020 06:05:46 GMT):
Currently I'm using one external LoadBalancer per k8s service that I want to access on my cluster (on DigitalOcean).
I.e. one LoadBalancer for each Peer and Orderer that should be accessible from outside the k8s cluster.
However, I want to move away from this complex and costly solution.
My goal is to use one external LoadBalancer per k8s cluster with an ingress controllers behind, which distributes the requests.
Unfortunately, I wasn't able to do that with nginx-ingress controller yet, maybe I'll have a look at different ingress-controllers.
ownspies (Mon, 24 Feb 2020 15:16:06 GMT):
I've tried to get it working with Traefik, however while Traefik supports gRPC, the AWS ALB does not. The ELB provides some, but not full support. Another issue that I've encountered is that none of the AWS LBs provide mTLS support.
ownspies (Mon, 24 Feb 2020 15:16:57 GMT):
I did open a ticket with AWS about the NLB issues we're seeing, apparently it is a known and very specific situation. The only work around is to make the NLB a public NLB and ensure the K8S worker nodes are in a private subnet, then use IP whitelisting in the NLB security group to limit access.
woodyjon (Mon, 24 Feb 2020 16:11:10 GMT):
Hello. Has anyone installed the hyperledger explorer in kubernetes next to a fabric network? I have a working Fabric v2.0 test-network running. But when I deploy the explorer pod, it has an error and exits. It seems to be a connection issue with the peer, but I have been playing around and cannot fix it. This is the logs from the explorer pod:
woodyjon (Mon, 24 Feb 2020 16:11:49 GMT):
```
false 'ssl-certs' '/opt/explorer/ssl-certs'
D0224 16:03:36.933652587 9 env_linux.cc:71] Warning: insecure environment read function 'getenv' used
D0224 16:03:37.606670676 9 env_linux.cc:71] Warning: insecure environment read function 'getenv' used
******* Initialization started for hyperledger fabric platform ******, { 'test-network': { name: 'test-network', profile: './test-network.json' } }
client_configs.name test-network client_configs.profile ./test-network.json
FabricUtils.createFabricClient
config.client.tlsEnable true
FabricConfig, this.config.channels mychannel
E0224 16:03:38.048102116 9 ssl_transport_security.cc:683] Invalid private key.
E0224 16:03:38.048203470 9 ssl_security_connector.cc:113] Handshaker factory creation failed with TSI_INVALID_ARGUMENT.
E0224 16:03:38.048229653 9 secure_channel_create.cc:152] Failed to create secure subchannel for secure name 'peer0-org1-example-com:7051'
E0224 16:03:38.048253174 9 secure_channel_create.cc:50] Failed to create channel args during subchannel creation.
2020-02-24T16:03:41.053Z - [31merror[39m: [Remote.js]: Error: Failed to connect before the deadline URL:grpcs://peer0-org1-example-com:7051
2020-02-24T16:03:41.055Z - [31merror[39m: [Channel.js]: Error: Failed to connect before the deadline URL:grpcs://peer0-org1-example-com:7051
E0224 16:03:41.064781812 9 ssl_transport_security.cc:683] Invalid private key.
E0224 16:03:41.064872844 9 ssl_security_connector.cc:113] Handshaker factory creation failed with TSI_INVALID_ARGUMENT.
E0224 16:03:41.064902519 9 secure_channel_create.cc:152] Failed to create secure subchannel for secure name 'peer0-org1-example-com:7051'
E0224 16:03:41.065032788 9 secure_channel_create.cc:50] Failed to create channel args during subchannel creation.
2020-02-24T16:03:44.065Z - [31merror[39m: [Remote.js]: Error: Failed to connect before the deadline URL:grpcs://peer0-org1-example-com:7051
********* call to initializeDetachClient **********
initializeDetachClient --> client_config { name: 'test-network', profile: './test-network.json' } name test-network
initializeDetachClient, network config) { name: 'test-network',
version: '1.0',
client:
{ tlsEnable: true,
adminUser: 'adminUser',
adminPassword: 'adminPassword',
enableAuthentication: false,
organization: 'Org1MSP',
connection: { timeout: [Object] } },
channels:
{ mychannel: { peers: [Object], orderers: [Object], connection: [Object] } },
organizations:
{ Org1MSP:
{ mspid: 'Org1MSP',
fullpath: true,
adminPrivateKey: [Object],
signedCert: [Object] },
Org2MSP:
{ mspid: 'Org2MSP',
fullpath: true,
adminPrivateKey: [Object],
signedCert: [Object] },
OrdererMSP:
{ mspid: 'OrdererMSP',
fullpath: true,
adminPrivateKey: [Object] } },
peers:
{ 'peer0-org1-example-com':
{ tlsCACerts: [Object],
url: 'grpcs://peer0-org1-example-com:7051',
eventUrl: 'grpcs://peer0-org1-example-com:7053',
grpcOptions: [Object] },
'peer0-org2-example-com':
{ tlsCACerts: [Object],
url: 'grpcs://peer0-org2-example-com:7051',
eventUrl: 'grpcs://peer0-org2-example-com:7053',
grpcOptions: [Object] } },
orderers:
{ 'orderer-example-com':
{ tlsCACerts: [Object],
url: 'grpcs://orderer-example-com:7050',
eventUrl: 'grpcs://orderer-example-com:7050',
grpcOptions: [Object] } } }
************************************* initializeDetachClient *************************************************
Error : Failed to connect client peer, please check the configuration and peer status
Info : Explorer will continue working with only DB data
************************************** initializeDetachClient ************************************************
FabricUtils.createDetachClient
Please open web browser to access ：http://localhost:8080/
pid is 9
FabricConfig, this.config.channels mychannel
<<<<<<<<<<<<<<<<<<<<<<<<<< Explorer Error >>>>>>>>>>>>>>>>>>>>>
Error : [ 'Default client peer is down and no channel details available database' ]
Received kill signal, shutting down gracefully
<<<<<<<<<<<<<<<<<<<<<<<<<< Closing explorer >>>>>>>>>>>>>>>>>>>>>
Closed out connections
```
woodyjon (Mon, 24 Feb 2020 16:12:12 GMT):
this is my explorer network config file:
woodyjon (Mon, 24 Feb 2020 16:12:31 GMT):
```
{
"name": "test-network",
"version": "1.0",
"client": {
"tlsEnable": true,
"adminUser": "adminUser",
"adminPassword": "adminPassword",
"enableAuthentication": false,
"organization": "Org1MSP",
"connection": {
"timeout": {
"peer": {
"endorser": "300"
},
"orderer": "300"
}
}
},
"channels": {
"mychannel": {
"peers": {
"peer0-org1-example-com": {},
"peer0-org2-example-com": {}
},
"orderers": {
"orderer-example-com" : {}
},
"connection": {
"timeout": {
"peer": {
"endorser": "6000",
"eventHub": "6000",
"eventReg": "6000"
}
}
}
}
},
"organizations": {
"Org1MSP": {
"mspid": "Org1MSP",
"fullpath": true,
"adminPrivateKey": {
"path":
"/fabric/crypto-config/peerOrganizations/org1-example-com/users/Admin@org1-example-com/msp/keystore/priv_sk"
},
"signedCert": {
"path":
"/fabric/crypto-config/peerOrganizations/org1-example-com/users/Admin@org1-example-com/msp/signcerts/Admin@org1-example-com-cert.pem"
}
},
"Org2MSP": {
"mspid": "Org2MSP",
"fullpath": true,
"adminPrivateKey": {
"path":
"/fabric/crypto-config/peerOrganizations/org2-example-com/users/Admin@org2-example-com/msp/keystore/priv_sk"
},
"signedCert": {
"path":
"/fabric/crypto-config/peerOrganizations/org2-example-com/users/Admin@org2-example-com/msp/signcerts/Admin@org2-example-com-cert.pem"
}
},
"OrdererMSP": {
"mspid": "OrdererMSP",
"fullpath": true,
"adminPrivateKey": {
"path":
"/fabric/crypto-config/ordererOrganizations/example-com/users/Admin@example-com/msp/keystore/priv_sk"
}
}
},
"peers": {
"peer0-org1-example-com": {
"tlsCACerts": {
"path":
"/fabric/crypto-config/peerOrganizations/org1-example-com/peers/peer0-org1-example-com/tls/ca.crt"
},
"url": "grpcs://peer0-org1-example-com:7051",
"eventUrl": "grpcs://peer0-org1-example-com:7053",
"grpcOptions": {
"ssl-target-name-override": "peer0-org1-example-com"
}
},
"peer0-org2-example-com": {
"tlsCACerts": {
"path":
"/fabric/crypto-config/peerOrganizations/org2-example-com/peers/peer0-org2-example-com/tls/ca.crt"
},
"url": "grpcs://peer0-org2-example-com:7051",
"eventUrl": "grpcs://peer0-org2-example-com:7053",
"grpcOptions": {
"ssl-target-name-override": "peer0-org1-example-com"
}
}
},
"orderers": {
"orderer-example-com": {
"tlsCACerts": {
"path":
"/fabric/crypto-config/ordererOrganizations/example-com/orderers/orderer-example-com/tls/ca.crt"
},
"url": "grpcs://orderer-example-com:7050",
"eventUrl": "grpcs://orderer-example-com:7050",
"grpcOptions": {
"ssl-target-name-override": "orderer-example-com"
}
}
}
}
```
woodyjon (Mon, 24 Feb 2020 16:13:40 GMT):
the peers have services, and the peers appear to be able to connect with each other without issue, as I can install chaincode, invoke it,... This is what a peer service looks like:
woodyjon (Mon, 24 Feb 2020 16:14:02 GMT):
```
apiVersion: v1
kind: Service
metadata:
name: {{ $deploymentName }}
labels:
run: {{ $deploymentName }}
spec:
type: ClusterIP
selector:
name: {{ $deploymentName }}
ports:
- protocol: TCP
port: 7051
name: grpc
- protocol: TCP
port: 7053
name: events
- protocol: TCP
port: 5984
name: couchdb
```
woodyjon (Mon, 24 Feb 2020 16:14:19 GMT):
If you have an idea, that would help. Thanks!
woodyjon (Mon, 24 Feb 2020 16:16:55 GMT):
I tried with grpc, grpcs. I don't understand if I have to put grpcs in the connection profile file and keep grpc in the service port
ownspies (Mon, 24 Feb 2020 19:36:22 GMT):
I have it working but am using a custom Helm chart I wrote...my guess is that your network config is not 100% correct though
ownspies (Mon, 24 Feb 2020 19:37:27 GMT):
```apiVersion: v1
kind: ConfigMap
metadata:
labels:
{{ include "hlfExplorer.standardLabels" . | indent 4 }}
name: {{ template "hlfExplorer.fullname" . }}-env
data:
DATABASE_HOST: "{{ .Values.database.hostname }}"
DATABASE_PORT: "{{ .Values.database.port }}"
DATABASE_DATABASE: "{{ .Values.database.databaseName }}"
DATABASE_NAME: "{{ .Values.database.databaseName }}"
DISCOVERY_AS_LOCALHOST: "{{ .Values.discovery.asLocalhost }}"```
ownspies (Mon, 24 Feb 2020 19:37:58 GMT):
```apiVersion: v1
kind: ConfigMap
metadata:
labels:
{{ include "hlfExplorer.standardLabels" . | indent 4 }}
name: {{ template "hlfExplorer.fullname" . }}-scripts
data:
run.sh: |
#!/bin/sh
cd $EXPLORER_APP_PATH
env | sort | egrep -v '^(PEER|ORDERER|CA|COUCHDB)'
printf 'Starting Hyperledger Fabric Explorer\n'
cp /tmp/config/* $EXPLORER_APP_PATH/app/platform/fabric/
cp /tmp/connection/* $EXPLORER_APP_PATH/app/platform/fabric/connection-profile/
node $EXPLORER_APP_PATH/main.js
tail -f /dev/null```
ownspies (Mon, 24 Feb 2020 19:41:04 GMT):
```apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "hlfExplorer.fullname" . }}
labels:
{{ include "hlfExplorer.standardLabels" . | indent 4 }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app.kubernetes.io/name: {{ include "hlfExplorer.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
template:
metadata:
labels:
app.kubernetes.io/name: {{ include "hlfExplorer.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
annotations:
# Allows forcing a deploy
"timestamp": "{{ date "20060102150405" .Release.Time }}"
spec:
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
envFrom:
- configMapRef:
name: {{ template "hlfExplorer.fullname" . }}-env
{{- range .Values.extraConfigmaps }}
- configMapRef:
name: {{ . }}
{{- end }}
- secretRef:
name: {{ .Values.database.existingSecret }}
{{- range .Values.extraSecrets }}
- secretRef:
name: {{ . }}
{{- end }}
command:
- /scripts/run.sh
env:
{{- range $key, $value := .Values.env }}
- name: "{{ $key }}"
value: "{{ $value }}"
{{- end }}
{{- range .Values.envFromSecrets }}
- name: "{{ .envKey }}"
valueFrom:
secretKeyRef:
name: "{{ .existingSecret }}"
key: "{{ .secretKey }}"
{{- end }}
{{- if .Values.service.enabled }}
ports:
- name: http
containerPort: {{ .Values.service.port }}
protocol: TCP
{{ end }}
volumeMounts:
- name: explorer-config
mountPath: /tmp/config
readOnly: true
- name: hyperledger-connection-profile
mountPath: /tmp/connection
readOnly: true
- name: scripts
mountPath: /scripts
readOnly: true
{{- range .Values.extraConfigmapMounts }}
- name: {{ .name }}
mountPath: {{ .mountPath }}
{{- if .subPath }}
subPath: {{ .subPath }}
{{- end }}
readOnly: {{ .readOnly }}
{{- end }}
{{- range .Values.extraSecretMounts }}
- name: {{ .name }}
mountPath: {{ .mountPath }}
{{- if .subPath }}
subPath: {{ .subPath | default ""}}
{{- end }}
readOnly: {{ .readOnly }}
{{- end }}
{{- range .Values.extraVolumeMounts }}
- name: {{ .name }}
mountPath: {{ .mountPath }}
{{- if .subPath }}
subPath: {{ .subPath }}
{{- end }}
readOnly: {{ .readOnly }}
{{- end }}
livenessProbe: {{ toYaml .Values.livenessProbe | nindent 12 }}
readinessProbe: {{ toYaml .Values.readinessProbe | nindent 12 }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
volumes:
- name: explorer-config
secret:
secretName: "{{ .Values.explorer.config.existingSecret }}"
- name: hyperledger-connection-profile
secret:
secretName: "{{ .Values.hyperledger.profileConfig.existingSecret }}"
- name: scripts
configMap:
name: {{ template "hlfExplorer.fullname" . }}-scripts
defaultMode: 0555
{{- if .Values.extraVolumes }}
{{ toYaml .Values.extraVolumes | indent 8 }}
{{- end }}
{{- if .Values.extraConfigmapVolumes }}
{{ toYaml .Values.extraConfigmapVolumes | indent 8 }}
{{- end }}
{{- if .Values.extraSecretVolumes }}
{{ toYaml .Values.extraSecretVolumes | indent 8 }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
```
ownspies (Mon, 24 Feb 2020 19:41:18 GMT):
pretty standard stuff
Shubham-koli (Tue, 25 Feb 2020 04:37:26 GMT):
@lionelronaldo
Hi, you should take a look at this.
Shubham-koli (Tue, 25 Feb 2020 04:37:26 GMT):
@lionelronaldo
Hi, you should take a look at this.
https://github.com/APGGroeiFabriek/PIVT#cross-cluster-raft-network
woodyjon (Tue, 25 Feb 2020 07:19:58 GMT):
thanks, I'll check out your files
woodyjon (Tue, 25 Feb 2020 13:28:03 GMT):
I have a question about the CAs. When you deploy a ca on kubernetes, it will generate the tls files. How do you share the tls-cert.pem file with the peers so that they can connect with the CA?
ownspies (Tue, 25 Feb 2020 18:55:57 GMT):
Hey @woodyjon what I have done is generate the TLS files outside of the CA so our CA is not using it's own certificate; also you should distribute the TLS _chain_ not the cert itself
woodyjon (Wed, 26 Feb 2020 07:37:48 GMT):
Ok thanks. Ho man, this stuff is hard ;-)
ShrutiHK (Thu, 27 Feb 2020 10:14:21 GMT):
Hi All,
ShrutiHK (Thu, 27 Feb 2020 10:14:58 GMT):
I have deployed 5 orderers in raft and other peer nodes in pods
ShrutiHK (Thu, 27 Feb 2020 10:15:26 GMT):
while creating the channel, I am facing this error -
*error validating DeltaSet: policy for [Group] /Channel/Application not satisfied: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Admins' sub-policies to be satisfied*
ShrutiHK (Thu, 27 Feb 2020 10:16:40 GMT):
Requesting anyone who has faced this error to kindly help me out on this
ShrutiHK (Thu, 27 Feb 2020 10:17:20 GMT):
Also, have another query - Do we need to install go on the k8s cluster separately?
ownspies (Thu, 27 Feb 2020 13:14:47 GMT):
Inside the K8S worker nodes - no, inside the Fabric Peer nodes, also no
ownspies (Thu, 27 Feb 2020 13:16:50 GMT):
I've had this when the configtx.yaml does not have correct policies, so would suggest starting there... That said, I also get this for no good reason and simply regenerating the configs using configtx-gen fixed it magically. Our setup currently includes six channels and typically when I create them, two - three fail with this error; I simply regenerate the config and then re-run the create for the failed channels and voila, it works I have been wanting to open a bug report, but the I don't have an easy way to reproduce
ShrutiHK (Fri, 28 Feb 2020 07:25:19 GMT):
Okay, thanks
Mozer18 (Sun, 01 Mar 2020 11:44:57 GMT):
Has joined the channel.
ShrutiHK (Mon, 02 Mar 2020 07:20:00 GMT):
Hi @ownspies , this error seems to be gone now. I created the crypto materials and other artifacts using fabric 1.4.2 (earlier I was using 1.4.3).
ShrutiHK (Mon, 02 Mar 2020 07:20:27 GMT):
But, I get another error which I don't know what to do about -
ShrutiHK (Mon, 02 Mar 2020 07:20:49 GMT):
*[36m2020-03-02 07:12:20.620 UTC [grpc] createTransport -> DEBU 043[0m grpc: addrConn.createTransport failed to connect to {orderer:7050 0
ownspies (Mon, 02 Mar 2020 13:37:03 GMT):
IIRC that is displayed when you're using TLS mutual auth but don't have the right cert chain specified for CORE_PEER_TLS_CLIENTROOTCAS_FILES
lionelronaldo (Tue, 03 Mar 2020 08:50:14 GMT):
@Shubham-koli Wow thanks for letting me know, I wasn't aware that they updated the project! Thank you so much :pray: :pray: :rocket:
randyshu (Wed, 04 Mar 2020 07:05:52 GMT):
Has joined the channel.
vanitas92 (Mon, 09 Mar 2020 08:27:33 GMT):
Hi everyone, a colleague and I have made a post on how to implement the new feature of External Chaincodes in Kubernetes environments, take a look and happy to receive some feedback from it. Thanks! https://medium.com/@pau.aragones/how-to-implement-hyperledger-fabric-external-chaincodes-within-a-kubernetes-cluster-fd01d7544523
tengc (Thu, 12 Mar 2020 02:43:29 GMT):
Hello, I'm trying to install chaincode onto a 2.0 network and I encountered this error: Error: error getting chaincode deployment spec for transactive-energy: exit status 1. Does anyone have any idea what caused this?
Abhishekkishor (Thu, 12 Mar 2020 19:25:56 GMT):
Has joined the channel.
rahulhegde (Mon, 16 Mar 2020 20:54:08 GMT):
Has joined the channel.
guptasndp10 (Fri, 20 Mar 2020 13:44:14 GMT):
@vanitas92 I am trying out this external-chaincode feature with your repo but I am stuck at one issue where I am not able to invoke the transatcions. When I checked the peer logs it says error while intantitating the chaincode. The error is given below
2020-03-20T12:59:29.994123454Z [33m2020-03-20 12:59:29.993 UTC [lifecycle] Work -> WARN 081[0m
could not launch chaincode 'marbles:597ab5e37f30164ba2bd956015bfce6e9b94618390e026a7a844dfc0e06013ef':
connection to marbles:597ab5e37f30164ba2bd956015bfce6e9b94618390e026a7a844dfc0e06013ef failed:
error cannot create connection for marbles:597ab5e37f30164ba2bd956015bfce6e9b94618390e026a7a844dfc0e06013ef:
error creating grpc connection to chaincode-marbles-org1.hyperledger:7052: failed to create new connection:
connection error: desc = "transport: error while dialing: dial tcp: lookup chaincode-marbles-org1.hyperledger on 10.96.0.10:53: no such host"
vanitas92 (Fri, 20 Mar 2020 13:53:42 GMT):
Oh there is a bug in the service selectors of the k8s service i will fix it right away!
vanitas92 (Fri, 20 Mar 2020 13:58:33 GMT):
I ahve corrected the files and pushed them to the repository, just redeploy the chaincodes by issuing again the `kubectl create -f chaincode/k8s`, the new services will be up and running
vanitas92 (Fri, 20 Mar 2020 13:58:33 GMT):
I have corrected the files and pushed them to the repository, just redeploy the chaincodes by issuing again the `kubectl create -f chaincode/k8s`, the new services will be up and running
vanitas92 (Fri, 20 Mar 2020 13:59:17 GMT):
Thanks for letting me know!
guptasndp10 (Fri, 20 Mar 2020 14:05:23 GMT):
Thanks for quick response. I will try this and get back to you.
guptasndp10 (Fri, 20 Mar 2020 18:48:02 GMT):
I retried couple of times with the repo changes but getting some other error now with invocation. Please find below the peer logs which shows that even though init happened successfully but it says failed to invoke and advice to init chaincode before invoke
2020-03-20T18:37:58.088663884Z [34m2020-03-20 18:37:58.088 UTC [lifecycle] CheckCommitReadiness -> INFO 0cb[0m Successfully checked commit readiness of chaincode name 'marbles' on channel 'mychannel' with definition {sequence: 5, endorsement info: (version: '2.0', plugin: 'escc', init required: true), validation info: (plugin: 'vscc', policy: '0a2c120c120a080212020800120208011a0d120b0a076f7267314d535010031a0d120b0a076f7267324d53501003'), collections: (
vanitas92 (Sat, 21 Mar 2020 09:13:05 GMT):
hmmm i see there is a different sequence number, you have a value of 5. I have not tested it when upgrading or modifying this value so i cannot asess on that. However, i would make sure all peers have the same sequence and ccid of the chaincode so that signatures are valid. Just a guess here but if you find it out, please come here and share your experience!
javrevasandeep (Sat, 21 Mar 2020 10:09:40 GMT):
Finally I get it to working. Issue is in the invoke statement where the first time invocation requires --isInit flag but in documentation this flag was missing.
vanitas92 (Sat, 21 Mar 2020 10:12:56 GMT):
Ah true, missed it, i will fix it right now! Thanks!
vanitas92 (Sat, 21 Mar 2020 10:13:52 GMT):
Ah true! Completely missed it on the article. Going to add it now, thanks!!
vanitas92 (Sat, 21 Mar 2020 10:20:37 GMT):
Done, sorry for this issue!!
javrevasandeep (Sat, 21 Mar 2020 10:23:26 GMT):
No problem and thanks for your wonderful article. Do you have any other repos based on hyperledger fabric version 1.4 on kubernetes with mutual tls and node sdk
vanitas92 (Sat, 21 Mar 2020 10:28:31 GMT):
This was my first time publishing something like this so unfortunately i do not have published anything else. I have not tried the mutual TLS and node sdk but i have found this article, see if you find what you are looking for: https://medium.com/@thotanarendranathreddy/hyperledger-fabric-mutual-tls-6c0cdd8dc1c
ZainabM (Mon, 23 Mar 2020 05:44:09 GMT):
Is it open source or paid one?
vanitas92 (Mon, 23 Mar 2020 09:03:32 GMT):
The article is open, as well as the github code, and Hyperledger Fabric is open source as well!
przemyslaw.sanecki (Thu, 26 Mar 2020 14:00:55 GMT):
Has joined the channel.
przemyslaw.sanecki (Thu, 26 Mar 2020 14:05:38 GMT):
having trouble with new v2 chaincode lifecycle, I am using docker vm dind endpoint https://127.0.0.1 with tls on, peer has all docker client crypto material set
```
CORE_VM_DOCKER_TLS_ENABLED=true
CORE_VM_DOCKER_TLS_CERT=/tmp/org1/peer1/docker/cert.pem
CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=host
CORE_VM_DOCKER_TLS_KEY=/tmp/org1/peer1/docker/key.pem
CORE_VM_ENDPOINT=https://127.0.0.1:2376
CORE_VM_DOCKER_TLS_CA=/tmp/org1/peer1/docker/ca.pem
```
Trying to install chaincode package.
```
peer lifecycle chaincode install patient_consent-v0.0.1-package.tar.gz \
--peerAddresses fabric-dev-peer1-org1:7051 --connTimeout 10s \
--tlsRootCertFiles /tmp/org1/peer1/tls/msp/cacerts/fabric-dev-tlsca-org1-7052.pem \
-o fabric-dev-orderer1-org1:7050 --tls --cafile /tmp/org1/peer1/tls/msp/cacerts/fabric-dev-tlsca-org1-7052.pem
```
This gives me
```
Error: chaincode install failed with status: 500 - failed to invoke backing implementation of 'InstallChaincode': could not build chaincode: docker build failed: docker image inspection failed: Get https://127.0.0.1:2376/images/dev-peer1-org1-patient_consent-v0.0.1-9aedb4f5f58cb4bf18cf38f53751928caf9074c4bcb6859d8417fb37c09ab596-0acf342a6da8bfef85ec6b4d9dbe3ca4236ab9e52d903bb9fb014db836696d7b/json: remote error: tls: bad certificate
```
przemyslaw.sanecki (Thu, 26 Mar 2020 14:05:38 GMT):
having trouble with new v2 chaincode lifecycle, I am using docker vm dind endpoint https://127.0.0.1 with tls on, peer has all docker client crypto material set
```
CORE_VM_DOCKER_TLS_ENABLED=true
CORE_VM_DOCKER_TLS_CERT=/tmp/org1/peer1/docker/cert.pem
CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=host
CORE_VM_DOCKER_TLS_KEY=/tmp/org1/peer1/docker/key.pem
CORE_VM_ENDPOINT=https://127.0.0.1:2376
CORE_VM_DOCKER_TLS_CA=/tmp/org1/peer1/docker/ca.pem
```
Trying to install chaincode package.
```
peer lifecycle chaincode install patient_consent-v0.0.1-package.tar.gz \
--peerAddresses fabric-dev-peer1-org1:7051 --connTimeout 10s \
--tlsRootCertFiles /tmp/org1/peer1/tls/msp/cacerts/fabric-dev-tlsca-org1-7052.pem \
-o fabric-dev-orderer1-org1:7050 --tls --cafile /tmp/org1/peer1/tls/msp/cacerts/fabric-dev-tlsca-org1-7052.pem
```
This gives me
```
Error: chaincode install failed with status: 500 - failed to invoke backing implementation of 'InstallChaincode': could not build chaincode: docker build failed: docker image inspection failed: Get https://127.0.0.1:2376/images/dev-peer1-org1-patient_consent-v0.0.1-9aedb4f5f58cb4bf18cf38f53751928caf9074c4bcb6859d8417fb37c09ab596-0acf342a6da8bfef85ec6b4d9dbe3ca4236ab9e52d903bb9fb014db836696d7b/json: remote error: tls: bad certificate
```
Peer,cli and dind containers run in the same pod, docker certificates are valid as `docker --tlsverify ps` works as expected both from peer and cli containers
JayJong (Tue, 31 Mar 2020 09:18:47 GMT):
Hi all, im using kubernetes CA to approve the orderer's CSR but it seems like the orderer has an issue trying to create the CSR, the error is
` "message": "certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:anonymous\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope",`
i have checked that my orderer pod is in the same namespace as that was given the create rights. I had no issue with the kafka pod creating the CSR.
`kubectl create clusterrole cluster-edit-csr --resource=certificatesigningrequest --verb=create,delete,list,get`
`kubectl create clusterrolebinding cluster-edit-csr-bind01 --clusterrole=cluster-edit-csr --group=certificates.k8s.io --serviceaccount=orgorderer1:default`
JayJong (Tue, 31 Mar 2020 09:18:47 GMT):
Hi all, im using kubernetes CA to approve the orderer's CSR but it seems like the orderer has an issue trying to create the CSR, the error is
` "message": "certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:anonymous\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope",`
i have checked that my orderer pod is in the same namespace as that was given the create rights. I had no issue with the kafka pod creating the CSR.
Commands to enable usage of kubernetes CA and allow pods to submit request to kubernetes api server:
`kubectl create clusterrole cluster-edit-csr --resource=certificatesigningrequest --verb=create,delete,list,get`
`kubectl create clusterrolebinding cluster-edit-csr-bind01 --clusterrole=cluster-edit-csr --group=certificates.k8s.io --serviceaccount=orgorderer1:default`
woodyjon (Mon, 06 Apr 2020 13:20:45 GMT):
I am running fabric 2.0 on kubernetes. I deploy with helm. I have an issue deleting my deployment. Most of my pods stay in "terminated" state, and I have to manually force delete them to delete them. What graceful way do you use to quit your fabric 2.0 pods?
woodyjon (Mon, 06 Apr 2020 13:21:56 GMT):
I am running fabric 2.0 on kubernetes. I deploy with helm. I have an issue deleting my deployment. Most of my pods stay in "terminated" stat
pritam_01 (Mon, 06 Apr 2020 13:26:29 GMT):
Has joined the channel.
Adhavpavan (Tue, 07 Apr 2020 15:35:59 GMT):
Has joined the channel.
seokm0 (Fri, 10 Apr 2020 05:29:26 GMT):
Has joined the channel.
ZainabM (Wed, 15 Apr 2020 11:48:45 GMT):
Can anyone please tell me the best way to store application's wallet in kubernetes.
ZainabM (Wed, 15 Apr 2020 11:49:57 GMT):
ok..Thank you )
ZainabM (Wed, 15 Apr 2020 11:49:57 GMT):
ok..Thank you :)
vanitas92 (Fri, 24 Apr 2020 10:13:19 GMT):
Hi everyone, i would like to setup the peers 2.0 version in kubernetes yaml the CORE_CHAINCODE_EXTERNALBUILDERS through env vars but i am contslanty getting the following error Error: '': source data must be an array or slice, got string. I put the following input:
```
- name: CORE_CHAINCODE_EXTERNALBUILDERS
value: '[{name: golang-builder, path: /builders/golang}]'
```
I think that is because env var do not support arrays but i am not completely sure. The docs suggest to modify the core.yaml file itself but that does not have a good approach with kubernetes as it is the only config option to be modified in core.yaml file itself, though i have tried that and it works:
> Modify the chaincode stanza of the peer core.yaml file to include the externalBuilders configuration element:
```
externalBuilders:
- name: myexternal
path:
vanitas92 (Fri, 24 Apr 2020 10:13:19 GMT):
Hi everyone, i would like to setup the peers 2.0 version in kubernetes yaml the CORE_CHAINCODE_EXTERNALBUILDERS through env vars but i am contslanty getting the following error `Error: '': source data must be an array or slice, got string`. I put the following input:
```
- name: CORE_CHAINCODE_EXTERNALBUILDERS
value: '[{name: golang-builder, path: /builders/golang}]'
```
I think that is because env var do not support arrays but i am not completely sure. The docs suggest to modify the core.yaml file itself but that does not have a good approach with kubernetes as it is the only config option to be modified in core.yaml file itself, though i have tried that and it works:
> Modify the chaincode stanza of the peer core.yaml file to include the externalBuilders configuration element:
```
externalBuilders:
- name: myexternal
path:
cmgabriel (Wed, 29 Apr 2020 01:24:40 GMT):
Hello All - I just posted a public github repo with tutorial and youtube video for how to deploy a Hyperledger Fabric Certificate Authority on Kubernetes and interact with it to register and enroll identities, modify identities, and certificate management. Check it out on GitHub if you want https://github.com/denali49/fabric-ca-k8s
sureshappana (Wed, 29 Apr 2020 19:46:09 GMT):
Has joined the channel.
sureshappana (Wed, 29 Apr 2020 19:46:12 GMT):
Hi, I am trying to run HLF in k8s cluster, I am at the instantiation step. Getting error *Error: could not send: EOF*, I see chaincode container started successfully (and also runs without quitting/restarting). But after few minutes peer is getting the above error message.
I have attached peer logs below
Running HLF 1.4.6
Using golang chaincode (chaincode_example02) from fabric-samples
sureshappana (Wed, 29 Apr 2020 19:46:15 GMT):
peer logs.txt
AmanAgrawal (Mon, 04 May 2020 13:37:49 GMT):
Has joined the channel.
AmanAgrawal (Wed, 06 May 2020 10:36:40 GMT):
Hi All,
I am trying to setup fabric network on kubernetes using existing cryto certs that i have from my previous work. I have migrated those certs to my kubernetes setup. While i try to create channel with my existing tlscacerts, i get error on my orderer logs saying this: ServerHandshake -> ERRO 024[0m TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=10.28.2.11:40992
Any idea, if migrating the certs to kubernetes is not expected and if i am required to generate new certs on premise?
any thoughts on this
AmanAgrawal (Wed, 06 May 2020 10:36:40 GMT):
Hi All,
I am trying to setup fabric network on kubernetes using existing cryto certs that i have from my previous work. I have migrated those certs to my kubernetes setup. While i try to create channel with my existing tlscacerts, i get error on my orderer logs saying this: ServerHandshake -> ERRO 024[0m TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=10.28.2.11:40992
Any idea, if migrating the certs to kubernetes is not expected and if i am required to generate new certs on premise?
any thoughts on this @iramiller
AmanAgrawal (Wed, 06 May 2020 10:36:40 GMT):
Hi All,
I am trying to set up fabric network on Kubernetes using existing crypto certs that i have from my previous work. I have migrated those certs to my kubernetes setup. While i try to create channel with my existing tlscacerts, i get error on my orderer logs saying this: ServerHandshake -> ERRO 024[0m TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=10.28.2.11:40992,
seems like the certs arent being accepted here.
Any idea, if migrating the certs to kubernetes is not expected and if i am required to generate new certs on premise?
any help around is highly appreciated.
any thoughts on this @iramiller
sureshappana (Tue, 19 May 2020 17:25:12 GMT):
Hi, I am trying to setup HLF in K8s, when I run instantiation command, its starting the chaincode container successfully. But the instantiation step is exiting with EOF. When I perform any query execution I am getting the following error:
Error: endorsement failure during query. response: status:500 message:"make sure the chaincode test_cc has been successfully instantiated and try again: chaincode test_cc not found"
sureshappana (Tue, 19 May 2020 17:25:12 GMT):
Hi, I am trying to setup HLF in K8s, when I run instantiation command, its starting the chaincode container successfully. But the instantiation step is exiting with EOF. When I perform any query execution I am getting the following error:
```
Error: endorsement failure during query. response: status:500 message:"make sure the chaincode test_cc has been successfully instantiated and try again: chaincode test_cc not found"
```
sureshappana (Tue, 19 May 2020 17:25:38 GMT):
query_logs.txt
sureshappana (Tue, 19 May 2020 17:25:41 GMT):
instantiation_logs.txt
AaronWilmoth (Sat, 23 May 2020 16:57:54 GMT):
Has joined the channel.
cryptopatrick (Tue, 26 May 2020 19:59:57 GMT):
Has joined the channel.
tusharson (Wed, 27 May 2020 13:47:39 GMT):
Has joined the channel.
tusharson (Wed, 27 May 2020 13:48:25 GMT):
Any updated docs, videos on deploying hyperledger fabric in kubernetes ?
y0zg (Wed, 27 May 2020 18:47:38 GMT):
Has joined the channel.
y0zg (Wed, 27 May 2020 18:50:31 GMT):
We are planning to run single blockchain deployed on single k8s cluster per one customer. The next step is to have distributed communication between many customers with their own k8s clusters and deployed blockchains there. How do you guy perform the following scenario? I'm mostly interested in authentication mechanisms used between k8s clusters
y0zg (Wed, 27 May 2020 18:50:31 GMT):
We are planning to run single blockchain deployed on single k8s cluster per one customer. The next step is to have distributed communication between many customers with their own k8s clusters and deployed blockchains there. How do you guys perform the following scenario? I'm mostly interested in authentication mechanisms used between k8s clusters . Do you use any service mesh/multi-cloud/VPN/etc?
kim0 (Tue, 02 Jun 2020 15:26:47 GMT):
Has joined the channel.
kim0 (Tue, 02 Jun 2020 15:26:48 GMT):
Hello folks .. is there a maintained way to install Fabric on kubernetes ? like a helm chart or similar
cryptopatrick (Wed, 03 Jun 2020 07:15:56 GMT):
@kim0 Not sure if this can be of any help? https://youtu.be/ubrA3W1JMk0
AmanAgrawal (Fri, 05 Jun 2020 04:05:25 GMT):
Hi All,
AmanAgrawal (Fri, 05 Jun 2020 04:05:25 GMT):
Hi All, I need help understanding how can i mount my local storage to the volume mount inside my pod?
mrudav.shukla (Fri, 05 Jun 2020 05:44:56 GMT):
Check your orderer node logs as well. Is your instantiation transaction log successfully broadcasted and committed?
FarhanShafiq (Tue, 09 Jun 2020 07:23:27 GMT):
Has joined the channel.
lionelronaldo (Tue, 09 Jun 2020 16:33:18 GMT):
Hey guys!
As you probably realized, there are far more people needing help with HF + K8s (Hyperledger Fabric + Kubernetes) than there are people who have the knowledge and time to help. :astonished:
Just wanted to post again the amazingly awesome PIVT repository: https://github.com/APGGroeiFabriek/PIVT :fire: :rocket:
They have ready to use scripts and config files, and a README that walks you through every step. :person_climbing:
They offer production-ready setups for Kafka, RAFT, TLS, adding Orgs and even Cross K8s Cluster! :exploding_head:
This repo is open source and single handedly saves the HF K8s community. Huge thank you to these Devs! :heart_eyes:
Check it out, give it a star, learn some argo basics, and send the guys a donation/your gratitude! :pray: :shamrock:
sillysachin (Thu, 11 Jun 2020 14:30:25 GMT):
Is there any k8s and HF2.1 example , all the ones i have come across including PIVT are in 1.4.x
lionelronaldo (Sun, 14 Jun 2020 12:06:35 GMT):
just replace the container version with 2.0 and replace the old chaincode lifecycle with the new one
jkalwar (Tue, 30 Jun 2020 19:11:21 GMT):
Has joined the channel.
nekia (Thu, 02 Jul 2020 05:14:49 GMT):
Has joined the channel.
ps.agboola.ayodeji (Sun, 05 Jul 2020 09:46:51 GMT):
Has joined the channel.
matkt (Mon, 06 Jul 2020 21:13:48 GMT):
Has joined the channel.
anish-edx (Fri, 10 Jul 2020 05:34:54 GMT):
Has joined the channel.
mrudav.shukla (Sun, 12 Jul 2020 15:06:59 GMT):
Does AWS EKS CSI driver for EBS supports volume cloning?
julian (Mon, 27 Jul 2020 11:23:01 GMT):
Hello. I have setup a fabric on AWS EKS, version: v1.15.11-eks-af3caf. I have created a channel, and joined multiple peers. Next I tried to deploy a chaincode package using peer lifecycle chaincode install, but it fails with the following message: "2020-07-27 10:14:26.622 UTC [dockercontroller] buildImage -> ERRO f85 Error building image: docker build failed: Error returned from build: 1..." The full error message is here: https://pastebin.com/C2Ju906w
This looks like a name resolution error. Pods on the EKS cluster have connectivity to the internet. From the peer I'm able to download https://registry.npmjs.org/fabric-shim/-/fabric-shim-2.0.0.tgz without issue.
Looking through this Rocket.Chat channel I see others using IBM's K8 offering mention docker is not longer used as the container runtime, and docker in docker is required. Could this be the same for AWS EKS?
julian (Mon, 27 Jul 2020 11:23:01 GMT):
Hello. I have setup a 2.1.1 fabric on AWS EKS, version: v1.15.11-eks-af3caf. I have created a channel, and joined multiple peers. Next I tried to deploy a chaincode package using peer lifecycle chaincode install, but it fails with the following message: "2020-07-27 10:14:26.622 UTC [dockercontroller] buildImage -> ERRO f85 Error building image: docker build failed: Error returned from build: 1..." The full error message is here: https://pastebin.com/C2Ju906w
This looks like a name resolution error. Pods on the EKS cluster have connectivity to the internet. From the peer I'm able to download https://registry.npmjs.org/fabric-shim/-/fabric-shim-2.0.0.tgz without issue.
Looking through this Rocket.Chat channel I see others using IBM's K8 offering mention docker is not longer used as the container runtime, and docker in docker is required. Could this be the same for AWS EKS?
julian (Tue, 28 Jul 2020 14:23:19 GMT):
After further investigation I can see a chaincode container is being created on one of the EKS nodes. Therefore it's not a container runtime issue. I have inspected the chaincode container which starts and falls over, and I see the following config: https://pastebin.com/rEE4wfjA
It looks like the network mode is incorrect: "NetworkMode": "default". Previously in earlier version of fabric I had used the peer env var CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE to set this as required. It doesn't appear to be working this time around. I have attempted to set this to none, but it still shows as default when a chaincode container is started. Can anyone help?
joy_2_code (Sat, 01 Aug 2020 04:53:46 GMT):
Has joined the channel.
vanitas92 (Mon, 03 Aug 2020 14:18:01 GMT):
Hi Julian, I would not suggest to use the docker daemon from the EKS workers nodes themselves, as you will have no control about the chaincodes within the EKS domain. I would suggest to use the Docker-in-Docker approach if you are still using the legacy chaincode lifecycle, which i would not also recommend as it is currently deprecated. As you are using version 2.1.1, i would suggest using the new lifecycle chaincode installation and build your chaincodes as docker containers that are available with EKS domain. See the following documentation: https://hyperledger-fabric.readthedocs.io/en/release-2.2/cc_service.html
julian (Tue, 04 Aug 2020 09:10:05 GMT):
@vanitas92 Hello. I had seen the external service option, but I haven't researched it. I will read about it now. Thank you for your reply.
vanitas92 (Tue, 04 Aug 2020 09:11:37 GMT):
I have made a tutorial about this if you need further help with some examples and code templates https://medium.com/swlh/how-to-implement-hyperledger-fabric-external-chaincodes-within-a-kubernetes-cluster-fd01d7544523
julian (Tue, 04 Aug 2020 09:46:02 GMT):
Brilliant. Thank you.
julian (Tue, 04 Aug 2020 17:10:46 GMT):
@vanitas92 On further reading I see the chaincode as an external service is currently only available for go. Our chaincode is written in node. I will probably have to take the docker in docker route.
vanitas92 (Wed, 05 Aug 2020 14:15:34 GMT):
Yes, that might be the best approach for non golang based chaincodes for now, until they support the external chaincode feature :)
aatkddny (Fri, 07 Aug 2020 15:57:44 GMT):
anyone here using a standalone setup with raft and dind (better - with the java sdk).
have a new error (posted it in fabric-java-sdk) but thought it might have happened somewhere else using a different language.
go chaincode is failing to upgrade. this is in a long standing network with multiple prior successful upgrades. logs are inconclusive.
i'm thinking it's a resource issue but can't see anything obviously awry. anyone have this happen to them?
aatkddny (Fri, 07 Aug 2020 15:57:44 GMT):
anyone here using a standalone - multi-node but our datacenter standalone - setup with raft and dind (better - with the java sdk).
have a new error (posted it in fabric-java-sdk) but thought it might have happened somewhere else using a different language.
go chaincode is failing to upgrade. this is in a long standing network with multiple prior successful upgrades. logs are inconclusive.
i'm thinking it's a resource issue but can't see anything obviously awry. anyone have this happen to them?
jcldnatv (Sat, 22 Aug 2020 17:24:05 GMT):
Has joined the channel.
charyorde (Tue, 25 Aug 2020 14:29:39 GMT):
Has joined the channel.
bjwswang (Thu, 27 Aug 2020 02:12:20 GMT):
Has left the channel.
sstone1 (Thu, 03 Sep 2020 07:14:54 GMT):
Has left the channel.
AbhijeetSamanta (Mon, 14 Sep 2020 12:58:14 GMT):
I have implemented the Raft network with TLS with three orderer and installed and init Go chaincode however when I am trying to invoke chaincode its giving me error as
AbhijeetSamanta (Mon, 14 Sep 2020 12:58:14 GMT):
I have implemented the Raft network with TLS with three orderer and installed and init Go chaincode however when I am trying to invoke chaincode its giving me error as `[Channel.js]: Channel:common received discovery error:access denied
2020-09-13T21:56:11.992Z - error: [Channel.js]: Error: Channel:common Discovery error:access denied
2020-09-13T21:56:11.992Z - error: [Network]: _initializeInternalChannel: Unable to initialize channel. Attempted to contact 1 Peers. Last error was Error: Channel:common Discovery error:access denied`
popopame (Mon, 14 Sep 2020 17:08:33 GMT):
Has joined the channel.
vioking (Fri, 18 Sep 2020 16:51:51 GMT):
Has joined the channel.
gen_el (Fri, 25 Sep 2020 15:54:58 GMT):
Has joined the channel.
czar0 (Fri, 09 Oct 2020 11:11:15 GMT):
Has joined the channel.
skulos (Mon, 12 Oct 2020 13:13:03 GMT):
Has joined the channel.
c0deh0use (Thu, 15 Oct 2020 06:46:39 GMT):
Has joined the channel.
c0deh0use (Thu, 15 Oct 2020 07:17:44 GMT):
Hello guys, I'm successfully running HLF 2.2.1 locally with docker & docker-compose but I would like to migrate it k8s. I have problems establishing connection from peers to a single raft orderer. I was trying to use the IBM https://github.com/IBM/blockchain-network-on-kubernetes. First question I have is are you all using TLS internally in k8s network this examples does not set that option when creating the genesis.block and channel tx.
vanitas92 (Thu, 15 Oct 2020 13:50:03 GMT):
You can take a look at my repository on setting up a k8s based hyperledger fabric network with external chaincode feature, i think it also includes your answer on how you can generate TLS artefacts that match the inner services of K8s, although it is not the center theme of the article. See if that helps: https://medium.com/swlh/how-to-implement-hyperledger-fabric-external-chaincodes-within-a-kubernetes-cluster-fd01d7544523
vanitas92 (Thu, 15 Oct 2020 13:50:03 GMT):
You can take a look at my article on setting up a k8s based hyperledger fabric network with external chaincode feature, i think it also includes your answer on how you can generate TLS artefacts that match the inner services of K8s, although it is not the center theme of the article. See if that helps: https://medium.com/swlh/how-to-implement-hyperledger-fabric-external-chaincodes-within-a-kubernetes-cluster-fd01d7544523
tennenjl (Thu, 15 Oct 2020 19:00:09 GMT):
Has joined the channel.
nleut (Fri, 16 Oct 2020 20:34:28 GMT):
all of the logs from orderers, peers, couchdb, docker-in-docker containers have suddenly stopped appearing on my cluster. the istio-proxy containers in the same pod are still displaying logs and the cluster has been upgraded to latest stable version, nodes restarted. Any idea what might be causing this? there was no change in the configuration since the logs were working earlier
heena066 (Mon, 02 Nov 2020 09:29:14 GMT):
Has joined the channel.
shivraj (Thu, 05 Nov 2020 06:28:10 GMT):
Hey guys, I'm getting error logs saying " http: TLS handshake error from 192.168.219.40:28758: EOF " from TLS enabled peer's and orderer's deployments in Azure's AKS and AWS EKS, they are exposed using loadbalancer's . It will be helpfully if you guys give any suggestion to suppress or resolve this issue.
ygnr (Thu, 12 Nov 2020 10:40:43 GMT):
Can we run Hyperledger Fabric Peer V2.2 as non root user? Any one tried to run in kubernetes with security context set as to not to run as root? Also we are trying to run as external chaincode container.
cynicalsnail (Wed, 18 Nov 2020 07:01:27 GMT):
Has joined the channel.
melroy27 (Mon, 23 Nov 2020 05:26:49 GMT):
Has joined the channel.
c0deh0use (Mon, 23 Nov 2020 18:28:10 GMT):
Thanks Vanitas,
I've managed to us the PIVT repo and adopt it to use in Fabric 2.2.
I noticed you are using FabricCA. How are you getting the certificates needed to setup the communication with gateways?
c0deh0use (Mon, 23 Nov 2020 18:45:24 GMT):
How are you guys setting up FabricCa and all the certs of known users ?
melroy27 (Tue, 24 Nov 2020 10:20:00 GMT):
Hey guys i am new to Fabric but as of some requirement I am stuck at a point and looking for help ```
*The setup below is all done on Google Cloud.*
Setup 1: Docker
I have a Docker setup running on a VM instance having a Hyperledger Fabric Network(1.4.2) of 3 Orderers(etcdraft) and 2 peers(peer0, peer1) of a single organization.
As part of Docker setup the crypto-material generated is using Fabric CA. The channel that i have on the network is "mychannel"
Setup 2: Kubernetes
I have a Kubernetes Cluster running on GKE having just 1 peer pod(peer2).
The setup files realted to this pod are:
1. peer2 Deployment File
2. peer2 Service File
3. peer2 CLI
4. peer2 CouchDB
What I want to achieve is:
I want the peer pod of Kubernetes to be able to communicate with the network setup in Docker.
i.e. I want atleast 1 peer of the Kubernetes should be able to join the network channel(mychannel) on the Docker Network and be able to perform some transaction/query
like instantiation of chaincode and query the same.
Till now what I have done is exposed the services of the peer pod in Kubernetes i.e. ClusterIP -> NodePort.
Any help would be appreciated.
```
c0deh0use (Tue, 24 Nov 2020 11:06:07 GMT):
I'm not sure you can mix K8s cluster network with a single container.
archanaarige (Tue, 24 Nov 2020 11:25:31 GMT):
Has joined the channel.
c0deh0use (Wed, 25 Nov 2020 12:14:50 GMT):
Anyone could share how you guys are setting Fabric-CA ?
rameshlohala (Wed, 02 Dec 2020 12:12:00 GMT):
Has joined the channel.
HansrajRami (Fri, 04 Dec 2020 21:19:15 GMT):
Has joined the channel.
HansrajRami (Fri, 04 Dec 2020 21:24:07 GMT):
docker pull hansrajrami/fabric-peer:2.2.1-rootless
Run this
ZainabM (Mon, 07 Dec 2020 06:44:27 GMT):
When I start the hyper ledger network on a free Kubernetes classic cluster of IBM Cloud following the GitHub link https://github.com/IBM/blockchain-network-on-kubernetes.git, the chain code instantiates.
But when I try to instantiate the chain code in IBM's VPC cluster, I am getting Post http://docker:2375/containers/create? i/o timeout
Can anyone please help?
ravishankar.gu (Fri, 11 Dec 2020 08:21:33 GMT):
Has joined the channel.
adarshaJha (Fri, 18 Dec 2020 12:20:47 GMT):
HI suddenly
all of my tlsca certs expired
my network is on production
what to do in order to start the network from same state
how to renew my tls certs ?
my orderer and peers all are giving bad certificate error
HansrajRami (Tue, 22 Dec 2020 15:51:34 GMT):
For peers issue new TLS certificates from the same CA which issued their current certs
replace the existing certificates (or update core.yaml to point to the new certificates if using different paths/filenames)
restart the peer(s)
For orderers using Raft, what you can do is for orderers is use their existing private keys to generate new CSRs and then have the CA issue a new public certificate. You will then replace / update orderer.yaml the public certificate and restart the orderers one at a time.
HansrajRami (Tue, 22 Dec 2020 15:53:46 GMT):
For peers issue new TLS certificates from the same CA which issued their current certs
replace the existing certificates (or update core.yaml to point to the new certificates if using different paths/filenames)
restart the peer(s)
For orderers using Raft, what you can do is for orderers is use their existing private keys to generate new CSRs and then have the CA issue a new public certificate. You will then replace / update orderer.yaml the public certificate and restart the orderers one at a time.
HansrajRami (Tue, 22 Dec 2020 15:53:46 GMT):
This may help
yixinhuo (Tue, 22 Dec 2020 16:22:34 GMT):
Has joined the channel.
iLico (Sun, 27 Dec 2020 09:56:38 GMT):
Has joined the channel.
lupass93 (Tue, 29 Dec 2020 23:37:43 GMT):
Has joined the channel.
lupass93 (Tue, 29 Dec 2020 23:41:27 GMT):
Hi!,
I follow more resource found in Internet on chaincode as external builder, but I get a error when try to install the chaincode: Unknown chaincodeType: EXTERNAL.
Have you any idea for the causa of this error?
I set correctly core.yaml in cli container with the buildpack provided by fabric-samples
Please Help me!
rahulkundani (Thu, 31 Dec 2020 11:49:21 GMT):
Has joined the channel.
rahulkundani (Thu, 31 Dec 2020 11:49:22 GMT):
Hi all
I am newbie here I want to setup a production grade HLF, I have tried the test networks but I am having no knowledge about the production network can someone guide me to achieve my goal
Thanks
Sandyzhanghs (Sun, 10 Jan 2021 05:02:00 GMT):
Has joined the channel.
HansrajRami (Sun, 10 Jan 2021 17:15:45 GMT):
Hi all !!!
I am using fabric external chaincode with version 2.2.1
Network is working fine
But while invoking chaincode facing this error
2021-01-10 15:01:42.532 UTC [endorser] SimulateProposal -> ERRO 0a0 failed to invoke chaincode samplecc, error: fork/exec /etc/hyperledger/fabric/chaincode-buildpack/bin/run: no such file or directory
builder 'chaincode-buildpack' run failed to start
I have done this setup before without run script and it was working
I've tried putting run script at given path and restarted peer still having same issue
Any help will be appreciated
ygnr (Tue, 12 Jan 2021 22:31:54 GMT):
Anyone running Hyperledger Fabric on Azure Kubernetes (latest version 1.18.10+) and facing any issues? I have raised a bug here (https://jira.hyperledger.org/projects/FAB/issues/FAB-18396?filter=allopenissues). Microsoft said its something to do with Kernal
gentios (Mon, 18 Jan 2021 21:00:20 GMT):
Have someone upgraded a Fabric@1.4.x production kubernetes network to a Fabric@2.x, if so what would be the steps ?
awa (Sun, 31 Jan 2021 17:01:21 GMT):
Has joined the channel.
adityajoshi12 (Mon, 01 Feb 2021 04:56:48 GMT):
Has joined the channel.
BlockCrasher (Mon, 01 Feb 2021 05:51:55 GMT):
Has joined the channel.
adityajoshi12 (Wed, 03 Feb 2021 10:41:08 GMT):
Hey Folks, I have one doubt related to the deployment of my fabric network on kubernetes. I am planning to deploy my peers as statefulsets with 1 replica and in the near future i would be scaling them to 3 . I want to how the data replication between these statefulsets will happen and will scaling up work as they will not be in sync.
Unicow (Wed, 03 Feb 2021 14:14:41 GMT):
Has joined the channel.
Unicow (Thu, 04 Feb 2021 13:48:33 GMT):
Hi everyone,
Unicow (Thu, 04 Feb 2021 13:54:30 GMT):
Hi everyone,
I am trying to deploy HLF 2.2 on Kuberentes as part of a research project, but I have some issues with the configurations.
I started converting the docker-compose files from the test-network into k8s deployments, but there are a lot of configurations to create channels,join peers, not to mention installing CC plus pod to pod communication that has to be done makes things even more difficult.
Are there any configuration files or a guide how to do that?
I have already read about this article: https://kctheservant.medium.com/test-network-script-walk-through-95ca973bc676
but unfortunately it runs with errors on my machine after following the instructions at the point of starting the orderers:
e.g. `Hyperledger fabric:TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress`
Any help/hint is highly appreciated!
Thanks!
Unicow (Thu, 04 Feb 2021 13:54:30 GMT):
Hi everyone,
I am trying to deploy HLF 2.2 on Kuberentes as part of a research project, but I have some issues with the configurations.
I started converting the docker-compose files from the test-network into k8s deployments, but there are a lot of configurations to create channels,join peers, not to mention installing CC plus pod to pod communication that has to be done makes things even more difficult.
Are there any configuration files or a guide how to do that?
I have already read about this article: https://medium.com/swlh/how-to-implement-hyperledger-fabric-external-chaincodes-within-a-kubernetes-cluster-fd01d7544523
but unfortunately it runs with errors on my machine after following the instructions at the point of starting the orderers:
e.g. `Hyperledger fabric:TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress`
Any help/hint is highly appreciated!
Thanks!
AshutoshKumar7 (Thu, 11 Feb 2021 07:03:50 GMT):
Has joined the channel.
AshutoshKumar7 (Thu, 11 Feb 2021 07:05:32 GMT):
Please recommend approach / tool for deploying fabric on kubernetes - multiple clusters
vitorduarte (Fri, 19 Feb 2021 12:44:39 GMT):
Has joined the channel.
kartheekgottipati (Sat, 27 Feb 2021 07:49:12 GMT):
Has joined the channel.
Chem (Thu, 04 Mar 2021 11:26:16 GMT):
Has joined the channel.
Unicow (Sun, 07 Mar 2021 22:01:01 GMT):
Hello, I would like to deploy fabric ca server on Google Kubernetes Engine and the `fabric-ca-server-config.yaml` requires the `csr.hosts`. How can I get the hostname of the pod running in k8s cluster in order to put the hostname in the file?
```
csr:
cn: fabric-ca-server
keyrequest:
algo: ecdsa
size: 256
names:
- C: US
ST: "North Carolina"
L:
O: Hyperledger
OU: Fabric
hosts:
- localhost
```
ymo 7 (Fri, 19 Mar 2021 18:41:43 GMT):
Has joined the channel.
ymo 7 (Fri, 19 Mar 2021 18:42:15 GMT):
Anyone can recommend a fabric on k8s install of 2.3 ?
AbhijeetSamanta (Fri, 26 Mar 2021 05:07:21 GMT):
How to migrate data from IBM blockchain to other hyperledger fabric cloud? Please see question on stackoverflow https://stackoverflow.com/questions/66796313/how-to-migrate-data-from-ibm-blockchain-to-other-hyperledger-fabric-cloud
agustincharry (Tue, 06 Apr 2021 15:44:42 GMT):
Has joined the channel.
lukeledet (Thu, 08 Apr 2021 19:59:43 GMT):
Has joined the channel.
cmgabriel (Fri, 23 Apr 2021 23:34:24 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=E3HYbDtNfNFD8AphP) I have a tutorial video on YouTube on how to do this here: https://youtu.be/PbMxqH6bNB8
cmgabriel (Fri, 23 Apr 2021 23:36:04 GMT):
[ ](https://chat.hyperledger.org/channel/fabric-kubernetes?msg=mgNojYghE5ka2Mvye) I have done this and it works great. You will need to use the new peer chaincode lifecycle, and the new osnadmin commands. You do not need a system channel either.
HLFPOC (Thu, 29 Apr 2021 17:06:50 GMT):
Hi All,
I am trying to setup a fabric network (v2.3.1) in AWS EKS environment using chaincode as an external service but getting below error message after running the `peer lifecycle chaincode install` command:
Error:
```
Error: chaincode install failed with status: 500 - failed to invoke backing implementation of 'InstallChaincode': could not build chaincode: docker build failed: docker image inspection failed: Get "http://unix.sock/images/dev-peer0-org1-mycc -4e09ca97183ec70be708a4db7bf87295adf37c86f5d991ad82024a27f5df8d80-4794d06c41d224a533fdc93cfa93ed46abc5c4e679d0a803ad908a7e62ddfe54/json": dial unix /var/run/docker.sock: connect: no such file or directory
```
As per https://lists.hyperledger.org/g/fabric/topic/77402508?p=Created,,,20,2,0,0::recentpostdate%2Fsticky,,,20,2,0,77402508, it seems this error is because of some update by Docker desktop and it can be resolved by switching off the gRPC FUSE settings, but not able to figure out what needs to be done in case of managed kubernetes cluster (v1.19 EKS). Can anyone suggest how we can fix this error ?
ymo 7 (Fri, 30 Apr 2021 20:32:16 GMT):
You folks know what you need to do to get a peer cli running on your machine talking to an orderer/peer running in k8s via kubectl port forward when tls is on ? do i have to do anything special ? i am getting disconnection. and i see my localhost is listed in the logs of the peer but then the connection gets dropped
HansrajRami (Sun, 02 May 2021 19:00:10 GMT):
This happened with me as well few times
One reason can be that your build packs not having execute permission
And other that chaincode package in not packaged correctly so peer will assume that it normal chaincode build
Because if any of the configs doesn't match peer start building it as normal docker build
HansrajRami (Sun, 02 May 2021 19:08:28 GMT):
I am running Hyperledger fabric network on Kubernetes with ClusterIP services
Can anyone suggest how to use ingress with peer and orderer to access them outside of cluster?
ingjhon (Fri, 14 May 2021 18:28:13 GMT):
Has joined the channel.
sj1 4 (Sun, 30 May 2021 15:21:47 GMT):
Has joined the channel.
Unicow (Mon, 07 Jun 2021 13:37:04 GMT):
Hi all, are there any up-to-date helm charts to install hyperledger fabric on k8s?
Unicow (Thu, 10 Jun 2021 09:47:06 GMT):
Hi all,
I am trying to configure HLF to run on k8s. I have my orderer deployment but when I start the pod I get the following error:
```panic: unable to bootstrap orderer. Error reading genesis block file: read /var/hyperledger/orderer/orderer.genesis.block: is a directory```
Any idea what could be wrong?
Here is my orderer deployment:
```
apiVersion: apps/v1
kind: Deployment
metadata:
name: orderer1
namespace: hyperledger
spec:
selector:
matchLabels:
app: orderer1
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: orderer1
spec:
containers:
- args:
- sh
- -c
- orderer
env:
- name: FABRIC_LOGGING_SPEC
value: "INFO"
- name: ORDERER_GENERAL_LISTENADDRESS
value: 0.0.0.0
- name: ORDERER_OPERATIONS_LISTENADDRESS
value: 0.0.0.0:8443
- name: ORDERER_GENERAL_GENESISMETHOD
value: file
- name: ORDERER_GENERAL_GENESISFILE
value: /var/hyperledger/orderer/orderer.genesis.block
- name: ORDERER_GENERAL_LOCALMSPID
value: OrdererMSP
- name: ORDERER_GENERAL_LOCALMSPDIR
value: /var/hyperledger/orderer/msp
# enabled TLS
- name: ORDERER_GENERAL_TLS_ENABLED
value: "true"
- name: ORDERER_GENERAL_TLS_PRIVATEKEY
value: /var/hyperledger/orderer/tls/server.key
- name: ORDERER_GENERAL_TLS_CERTIFICATE
value: /var/hyperledger/orderer/tls/server.crt
- name: ORDERER_GENERAL_TLS_ROOTCAS
value: "[/var/hyperledger/orderer/tls/ca.crt]"
- name: ORDERER_KAFKA_TOPIC_REPLICATIONFACTOR
value: "1"
- name: ORDERER_KAFKA_VERBOSE
value: "true"
- name: ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE
value: /var/hyperledger/orderer/tls/server.crt
- name: ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY
value: /var/hyperledger/orderer/tls/server.key
- name: ORDERER_GENERAL_CLUSTER_ROOTCAS
value: "[/var/hyperledger/orderer/tls/ca.crt]"
workingDir: /opt/gopath/src/github.com/hyperledger/fabric
image: hyperledger/fabric-orderer:2.3
name: orderer1
ports:
- containerPort: 7050
- containerPort: 8443
volumeMounts:
- name: config-genesis
mountPath: /var/hyperledger/orderer/orderer.genesis.block
- name: config-msp
mountPath: /var/hyperledger/orderer/msp
- name: config-tls
mountPath: /var/hyperledger/orderer/tls
- name: config-keystore
mountPath: /var/hyperledger/orderer/msp/keystore/
- name: config-tlscacerts
mountPath: /var/hyperledger/orderer/tlscacerts
- name: config-cacerts
mountPath: /var/hyperledger/orderer/msp/cacerts/
- name: config-signcerts
mountPath: /var/hyperledger/orderer/msp/signcerts/
restartPolicy: Always
volumes:
- name: config-genesis
configMap:
name: genesis
- name: config-msp
configMap:
name: config-msp
- name: config-tls
configMap:
name: config-tls
- name: config-keystore
configMap:
name: config-keystore
- name: config-tlscacerts
configMap:
name: config-tlscacerts
- name: config-cacerts
configMap:
name: config-cacerts
- name: config-signcerts
configMap:
name: config-signcerts
---
apiVersion: v1
kind: Service
metadata:
name: orderer1
namespace: hyperledger
spec:
type: NodePort
selector:
app: orderer1
ports:
- protocol: TCP
port: 30004
nodePort: 30004
targetPort: 7050
```
Unicow (Thu, 10 Jun 2021 09:47:06 GMT):
Hi all,
I am trying to configure HLF to run on k8s. I have my orderer deployment but when I start the pod I get the following error:
`panic: unable to bootstrap orderer. Error reading genesis block file: read /var/hyperledger/orderer/orderer.genesis.block: is a directory`
Any idea what could be wrong?
Here is my orderer deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: orderer1
namespace: hyperledger
spec:
selector:
matchLabels:
app: orderer1
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: orderer1
spec:
containers:
- args:
- sh
- -c
- orderer
env:
- name: FABRIC_LOGGING_SPEC
value: "INFO"
- name: ORDERER_GENERAL_LISTENADDRESS
value: 0.0.0.0
- name: ORDERER_OPERATIONS_LISTENADDRESS
value: 0.0.0.0:8443
- name: ORDERER_GENERAL_GENESISMETHOD
value: file
- name: ORDERER_GENERAL_GENESISFILE
value: /var/hyperledger/orderer/orderer.genesis.block
- name: ORDERER_GENERAL_LOCALMSPID
value: OrdererMSP
- name: ORDERER_GENERAL_LOCALMSPDIR
value: /var/hyperledger/orderer/msp
# enabled TLS
- name: ORDERER_GENERAL_TLS_ENABLED
value: "true"
- name: ORDERER_GENERAL_TLS_PRIVATEKEY
value: /var/hyperledger/orderer/tls/server.key
- name: ORDERER_GENERAL_TLS_CERTIFICATE
value: /var/hyperledger/orderer/tls/server.crt
- name: ORDERER_GENERAL_TLS_ROOTCAS
value: "[/var/hyperledger/orderer/tls/ca.crt]"
- name: ORDERER_KAFKA_TOPIC_REPLICATIONFACTOR
value: "1"
- name: ORDERER_KAFKA_VERBOSE
value: "true"
- name: ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE
value: /var/hyperledger/orderer/tls/server.crt
- name: ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY
value: /var/hyperledger/orderer/tls/server.key
- name: ORDERER_GENERAL_CLUSTER_ROOTCAS
value: "[/var/hyperledger/orderer/tls/ca.crt]"
workingDir: /opt/gopath/src/github.com/hyperledger/fabric
image: hyperledger/fabric-orderer:2.3
name: orderer1
ports:
- containerPort: 7050
- containerPort: 8443
volumeMounts:
- name: config-genesis
mountPath: /var/hyperledger/orderer/orderer.genesis.block
- name: config-msp
mountPath: /var/hyperledger/orderer/msp
- name: config-tls
mountPath: /var/hyperledger/orderer/tls
- name: config-keystore
mountPath: /var/hyperledger/orderer/msp/keystore/
- name: config-tlscacerts
mountPath: /var/hyperledger/orderer/tlscacerts
- name: config-cacerts
mountPath: /var/hyperledger/orderer/msp/cacerts/
- name: config-signcerts
mountPath: /var/hyperledger/orderer/msp/signcerts/
restartPolicy: Always
volumes:
- name: config-genesis
configMap:
name: genesis
- name: config-msp
configMap:
name: config-msp
- name: config-tls
configMap:
name: config-tls
- name: config-keystore
configMap:
name: config-keystore
- name: config-tlscacerts
configMap:
name: config-tlscacerts
- name: config-cacerts
configMap:
name: config-cacerts
- name: config-signcerts
configMap:
name: config-signcerts
---
apiVersion: v1
kind: Service
metadata:
name: orderer1
namespace: hyperledger
spec:
type: NodePort
selector:
app: orderer1
ports:
- protocol: TCP
port: 30004
nodePort: 30004
targetPort: 7050
Unicow (Thu, 10 Jun 2021 09:47:06 GMT):
Hi all,
I am trying to configure HLF to run on k8s. I have my orderer deployment but when I start the pod I get the following error:
`panic: unable to bootstrap orderer. Error reading genesis block file: read /var/hyperledger/orderer/orderer.genesis.block: is a directory`
Any idea what could be wrong?
Here is my orderer deployment:
```
apiVersion: apps/v1
kind: Deployment
metadata:
name: orderer1
namespace: hyperledger
spec:
selector:
matchLabels:
app: orderer1
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: orderer1
spec:
containers:
- args:
- sh
- -c
- orderer
env:
- name: FABRIC_LOGGING_SPEC
value: "INFO"
- name: ORDERER_GENERAL_LISTENADDRESS
value: 0.0.0.0
- name: ORDERER_OPERATIONS_LISTENADDRESS
value: 0.0.0.0:8443
- name: ORDERER_GENERAL_GENESISMETHOD
value: file
- name: ORDERER_GENERAL_GENESISFILE
value: /var/hyperledger/orderer/orderer.genesis.block
- name: ORDERER_GENERAL_LOCALMSPID
value: OrdererMSP
- name: ORDERER_GENERAL_LOCALMSPDIR
value: /var/hyperledger/orderer/msp
# enabled TLS
- name: ORDERER_GENERAL_TLS_ENABLED
value: "true"
- name: ORDERER_GENERAL_TLS_PRIVATEKEY
value: /var/hyperledger/orderer/tls/server.key
- name: ORDERER_GENERAL_TLS_CERTIFICATE
value: /var/hyperledger/orderer/tls/server.crt
- name: ORDERER_GENERAL_TLS_ROOTCAS
value: "[/var/hyperledger/orderer/tls/ca.crt]"
- name: ORDERER_KAFKA_TOPIC_REPLICATIONFACTOR
value: "1"
- name: ORDERER_KAFKA_VERBOSE
value: "true"
- name: ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE
value: /var/hyperledger/orderer/tls/server.crt
- name: ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY
value: /var/hyperledger/orderer/tls/server.key
- name: ORDERER_GENERAL_CLUSTER_ROOTCAS
value: "[/var/hyperledger/orderer/tls/ca.crt]"
workingDir: /opt/gopath/src/github.com/hyperledger/fabric
image: hyperledger/fabric-orderer:2.3
name: orderer1
ports:
- containerPort: 7050
- containerPort: 8443
volumeMounts:
- name: config-genesis
mountPath: /var/hyperledger/orderer/orderer.genesis.block
- name: config-msp
mountPath: /var/hyperledger/orderer/msp
- name: config-tls
mountPath: /var/hyperledger/orderer/tls
- name: config-keystore
mountPath: /var/hyperledger/orderer/msp/keystore/
- name: config-tlscacerts
mountPath: /var/hyperledger/orderer/tlscacerts
- name: config-cacerts
mountPath: /var/hyperledger/orderer/msp/cacerts/
- name: config-signcerts
mountPath: /var/hyperledger/orderer/msp/signcerts/
restartPolicy: Always
volumes:
- name: config-genesis
configMap:
name: genesis
- name: config-msp
configMap:
name: config-msp
- name: config-tls
configMap:
name: config-tls
- name: config-keystore
configMap:
name: config-keystore
- name: config-tlscacerts
configMap:
name: config-tlscacerts
- name: config-cacerts
configMap:
name: config-cacerts
- name: config-signcerts
configMap:
name: config-signcerts
---
apiVersion: v1
kind: Service
metadata:
name: orderer1
namespace: hyperledger
spec:
type: NodePort
selector:
app: orderer1
ports:
- protocol: TCP
port: 30004
nodePort: 30004
targetPort: 7050
```
Unicow (Thu, 10 Jun 2021 09:47:06 GMT):
Hi all,
I am trying to configure HLF to run on k8s. I have my orderer deployment but when I start the pod I get the following error:
`panic: unable to bootstrap orderer. Error reading genesis block file: read /var/hyperledger/orderer/orderer.genesis.block: is a directory`
Any idea what could be wrong?
Here is my orderer deployment:
```
apiVersion: apps/v1
kind: Deployment
metadata:
name: orderer1
namespace: hyperledger
spec:
selector:
matchLabels:
app: orderer1
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: orderer1
spec:
containers:
- args:
- sh
- -c
- orderer
env:
- name: FABRIC_LOGGING_SPEC
value: "INFO"
- name: ORDERER_GENERAL_LISTENADDRESS
value: 0.0.0.0
- name: ORDERER_OPERATIONS_LISTENADDRESS
value: 0.0.0.0:8443
- name: ORDERER_GENERAL_GENESISMETHOD
value: file
- name: ORDERER_GENERAL_GENESISFILE
value: /var/hyperledger/orderer/orderer.genesis.block
- name: ORDERER_GENERAL_LOCALMSPID
value: OrdererMSP
- name: ORDERER_GENERAL_LOCALMSPDIR
value: /var/hyperledger/orderer/msp
# enabled TLS
- name: ORDERER_GENERAL_TLS_ENABLED
value: "true"
- name: ORDERER_GENERAL_TLS_PRIVATEKEY
value: /var/hyperledger/orderer/tls/server.key
- name: ORDERER_GENERAL_TLS_CERTIFICATE
value: /var/hyperledger/orderer/tls/server.crt
- name: ORDERER_GENERAL_TLS_ROOTCAS
value: "[/var/hyperledger/orderer/tls/ca.crt]"
- name: ORDERER_KAFKA_TOPIC_REPLICATIONFACTOR
value: "1"
- name: ORDERER_KAFKA_VERBOSE
value: "true"
- name: ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE
value: /var/hyperledger/orderer/tls/server.crt
- name: ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY
value: /var/hyperledger/orderer/tls/server.key
- name: ORDERER_GENERAL_CLUSTER_ROOTCAS
value: "[/var/hyperledger/orderer/tls/ca.crt]"
workingDir: /opt/gopath/src/github.com/hyperledger/fabric
image: hyperledger/fabric-orderer:2.3
name: orderer1
ports:
- containerPort: 7050
- containerPort: 8443
volumeMounts:
- name: config-genesis
mountPath: /var/hyperledger/orderer/orderer.genesis.block
- name: config-msp
mountPath: /var/hyperledger/orderer/msp
- name: config-tls
mountPath: /var/hyperledger/orderer/tls
- name: config-keystore
mountPath: /var/hyperledger/orderer/msp/keystore/
- name: config-tlscacerts
mountPath: /var/hyperledger/orderer/tlscacerts
- name: config-cacerts
mountPath: /var/hyperledger/orderer/msp/cacerts/
- name: config-signcerts
mountPath: /var/hyperledger/orderer/msp/signcerts/
restartPolicy: Always
volumes:
- name: config-genesis
configMap:
name: genesis
- name: config-msp
configMap:
name: config-msp
- name: config-tls
configMap:
name: config-tls
- name: config-keystore
configMap:
name: config-keystore
- name: config-tlscacerts
configMap:
name: config-tlscacerts
- name: config-cacerts
configMap:
name: config-cacerts
- name: config-signcerts
configMap:
name: config-signcerts
---
apiVersion: v1
kind: Service
metadata:
name: orderer1
namespace: hyperledger
spec:
type: NodePort
selector:
app: orderer1
ports:
- protocol: TCP
port: 30004
nodePort: 30004
targetPort: 7050
```
And I create a genesis.block with configmap:
```
kubectl create configmap genesis --from-file=artifacts/genesis.block --namespace hyperledger
```
Unicow (Thu, 10 Jun 2021 09:47:06 GMT):
Hi all,
I am trying to configure HLF to run on k8s. I have my orderer deployment but when I start the pod I get the following error:
`panic: unable to bootstrap orderer. Error reading genesis block file: read /var/hyperledger/orderer/orderer.genesis.block: is a directory`
Any idea what could be wrong?
Here is my orderer deployment:
```
apiVersion: apps/v1
kind: Deployment
metadata:
name: orderer1
namespace: hyperledger
spec:
selector:
matchLabels:
app: orderer1
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: orderer1
spec:
containers:
- args:
- sh
- -c
- orderer
env:
- name: FABRIC_LOGGING_SPEC
value: "INFO"
- name: ORDERER_GENERAL_LISTENADDRESS
value: 0.0.0.0
- name: ORDERER_OPERATIONS_LISTENADDRESS
value: 0.0.0.0:8443
- name: ORDERER_GENERAL_GENESISMETHOD
value: file
- name: ORDERER_GENERAL_GENESISFILE
value: /var/hyperledger/orderer/orderer.genesis.block
- name: ORDERER_GENERAL_LOCALMSPID
value: OrdererMSP
- name: ORDERER_GENERAL_LOCALMSPDIR
value: /var/hyperledger/orderer/msp
# enabled TLS
- name: ORDERER_GENERAL_TLS_ENABLED
value: "true"
- name: ORDERER_GENERAL_TLS_PRIVATEKEY
value: /var/hyperledger/orderer/tls/server.key
- name: ORDERER_GENERAL_TLS_CERTIFICATE
value: /var/hyperledger/orderer/tls/server.crt
- name: ORDERER_GENERAL_TLS_ROOTCAS
value: "[/var/hyperledger/orderer/tls/ca.crt]"
- name: ORDERER_KAFKA_TOPIC_REPLICATIONFACTOR
value: "1"
- name: ORDERER_KAFKA_VERBOSE
value: "true"
- name: ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE
value: /var/hyperledger/orderer/tls/server.crt
- name: ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY
value: /var/hyperledger/orderer/tls/server.key
- name: ORDERER_GENERAL_CLUSTER_ROOTCAS
value: "[/var/hyperledger/orderer/tls/ca.crt]"
workingDir: /opt/gopath/src/github.com/hyperledger/fabric
image: hyperledger/fabric-orderer:2.3
name: orderer1
ports:
- containerPort: 7050
- containerPort: 8443
volumeMounts:
- name: config-genesis
mountPath: /var/hyperledger/orderer/orderer.genesis.block
- name: config-msp
mountPath: /var/hyperledger/orderer/msp
- name: config-tls
mountPath: /var/hyperledger/orderer/tls
- name: config-keystore
mountPath: /var/hyperledger/orderer/msp/keystore/
- name: config-tlscacerts
mountPath: /var/hyperledger/orderer/tlscacerts
- name: config-cacerts
mountPath: /var/hyperledger/orderer/msp/cacerts/
- name: config-signcerts
mountPath: /var/hyperledger/orderer/msp/signcerts/
restartPolicy: Always
volumes:
- name: config-genesis
configMap:
name: genesis
- name: config-msp
configMap:
name: config-msp
- name: config-tls
configMap:
name: config-tls
- name: config-keystore
configMap:
name: config-keystore
- name: config-tlscacerts
configMap:
name: config-tlscacerts
- name: config-cacerts
configMap:
name: config-cacerts
- name: config-signcerts
configMap:
name: config-signcerts
---
apiVersion: v1
kind: Service
metadata:
name: orderer1
namespace: hyperledger
spec:
type: NodePort
selector:
app: orderer1
ports:
- protocol: TCP
port: 30004
nodePort: 30004
targetPort: 7050
```
And I create a configmap for the genesis.block:
```
kubectl create configmap genesis --from-file=artifacts/genesis.block --namespace hyperledger
```
Unicow (Thu, 10 Jun 2021 09:47:06 GMT):
Hi all,
I am trying to configure HLF to run on k8s. I have my orderer deployment but when I start the pod I get the following error:
`panic: unable to bootstrap orderer. Error reading genesis block file: read /var/hyperledger/orderer/orderer.genesis.block: is a directory`
Any idea what could be wrong?
Here is my orderer deployment:
```
apiVersion: apps/v1
kind: Deployment
metadata:
name: orderer1
namespace: hyperledger
spec:
selector:
matchLabels:
app: orderer1
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: orderer1
spec:
containers:
- args:
- sh
- -c
- orderer
env:
- name: FABRIC_LOGGING_SPEC
value: "INFO"
- name: ORDERER_GENERAL_LISTENADDRESS
value: 0.0.0.0
- name: ORDERER_OPERATIONS_LISTENADDRESS
value: 0.0.0.0:8443
- name: ORDERER_GENERAL_GENESISMETHOD
value: file
- name: ORDERER_GENERAL_GENESISFILE
value: /var/hyperledger/orderer/orderer.genesis.block
- name: ORDERER_GENERAL_LOCALMSPID
value: OrdererMSP
- name: ORDERER_GENERAL_LOCALMSPDIR
value: /var/hyperledger/orderer/msp
# enabled TLS
- name: ORDERER_GENERAL_TLS_ENABLED
value: "true"
- name: ORDERER_GENERAL_TLS_PRIVATEKEY
value: /var/hyperledger/orderer/tls/server.key
- name: ORDERER_GENERAL_TLS_CERTIFICATE
value: /var/hyperledger/orderer/tls/server.crt
- name: ORDERER_GENERAL_TLS_ROOTCAS
value: "[/var/hyperledger/orderer/tls/ca.crt]"
- name: ORDERER_KAFKA_TOPIC_REPLICATIONFACTOR
value: "1"
- name: ORDERER_KAFKA_VERBOSE
value: "true"
- name: ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE
value: /var/hyperledger/orderer/tls/server.crt
- name: ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY
value: /var/hyperledger/orderer/tls/server.key
- name: ORDERER_GENERAL_CLUSTER_ROOTCAS
value: "[/var/hyperledger/orderer/tls/ca.crt]"
workingDir: /opt/gopath/src/github.com/hyperledger/fabric
image: hyperledger/fabric-orderer:2.3
name: orderer1
ports:
- containerPort: 7050
- containerPort: 8443
volumeMounts:
- name: config-genesis
mountPath: /var/hyperledger/orderer/orderer.genesis.block
- name: config-msp
mountPath: /var/hyperledger/orderer/msp
- name: config-tls
mountPath: /var/hyperledger/orderer/tls
- name: config-keystore
mountPath: /var/hyperledger/orderer/msp/keystore/
- name: config-tlscacerts
mountPath: /var/hyperledger/orderer/tlscacerts
- name: config-cacerts
mountPath: /var/hyperledger/orderer/msp/cacerts/
- name: config-signcerts
mountPath: /var/hyperledger/orderer/msp/signcerts/
restartPolicy: Always
volumes:
- name: config-genesis
configMap:
name: genesis
- name: config-msp
configMap:
name: config-msp
- name: config-tls
configMap:
name: config-tls
- name: config-keystore
configMap:
name: config-keystore
- name: config-tlscacerts
configMap:
name: config-tlscacerts
- name: config-cacerts
configMap:
name: config-cacerts
- name: config-signcerts
configMap:
name: config-signcerts
---
apiVersion: v1
kind: Service
metadata:
name: orderer1
namespace: hyperledger
spec:
type: NodePort
selector:
app: orderer1
ports:
- protocol: TCP
port: 30004
nodePort: 30004
targetPort: 7050
```
And I create a configmap for the genesis.block:
```
kubectl create configmap genesis --from-file=artifacts/genesis.block --namespace hyperledger
```
Update
These are the contents of the `/var/hyperledger/orderer/` directory inside the pod:
```
drwxrwxrwx 6 root root 4096 Jun 10 10:06 msp
drwxrwxrwx 3 root root 4096 Jun 10 10:06 orderer.genesis.block
drwxrwxrwx 3 root root 4096 Jun 10 10:06 tls
drwxrwxrwx 3 root root 4096 Jun 10 10:06 tlscacerts
```
And the genesis.block file is `d`.
Unicow (Mon, 14 Jun 2021 08:39:01 GMT):
Hello, I'm trying setup k8s with HLF and while creating a channel I get the following error:
```
osnadmin channel join --channel-id=$CHANNEL_NAME --config-block ./channel-artifacts/${CHANNEL_NAME}.block -o 192.168.49.2:30004 --ca-file "$ORDERER_CA" --client-cert "$ORDERER_ADMIN_TLS_SIGN_CERT" --client-key "$ORDERER_ADMIN_TLS_PRIVATE_KEY"
Error: Post "https://192.168.49.2:30004/participation/v1/channels": x509: cannot validate certificate for 192.168.49.2 because it doesn't contain any IP SANs
```
Could you please help? The up is my cluster ip for now and 30004 is a nodeport svc that connects with the orderer pod
Unicow (Mon, 14 Jun 2021 08:39:01 GMT):
Hello, I'm trying setup k8s with HLF and while creating a channel I get the following error:
```
osnadmin channel join --channel-id=$CHANNEL_NAME --config-block ./channel-artifacts/${CHANNEL_NAME}.block -o 192.168.49.2:30004 --ca-file "$ORDERER_CA" --client-cert "$ORDERER_ADMIN_TLS_SIGN_CERT" --client-key "$ORDERER_ADMIN_TLS_PRIVATE_KEY"
Error: Post "https://192.168.49.2:30004/participation/v1/channels": x509: cannot validate certificate for 192.168.49.2 because it doesn't contain any IP SANs
```
Could you please help? The IP is my cluster ip and 30004 is a nodeport svc that connects with the orderer pod
alicia.antony (Fri, 25 Jun 2021 10:21:01 GMT):
Has joined the channel.
AbhijeetSamanta (Wed, 28 Jul 2021 10:46:12 GMT):
Hi Team, I am Getting error when sumbit transaction on hyperledger fabric 2.x in orderer as rejected by Consenter: channel mainchannel is not serviced by me on orderer however all orderer is running fine
please help me someone it
nkaramolegos (Thu, 30 Sep 2021 11:59:44 GMT):
Has joined the channel.
nkaramolegos (Thu, 30 Sep 2021 11:59:44 GMT):
Hello, for using hyperledger fabric v2.2.0 with kubernetes where do i start? Is there any ready examples for single node cluster developement?
nkaramolegos (Thu, 30 Sep 2021 11:59:44 GMT):
Hello, for using hyperledger fabric v2.2.0 with kubernetes where do i start? Is there any ready examples for single node cluster development?
RocMax (Fri, 01 Oct 2021 23:50:09 GMT):
Has joined the channel.
roclee (Mon, 04 Oct 2021 06:10:40 GMT):
I asked this question in #fabric-questions channel but I found that maybe this channel is more appropriate.
roclee (Mon, 04 Oct 2021 06:10:40 GMT):
Hello everyone.
I am deploying a fabric network to Kubernetes with external chaincode service. But I met a quite weird problem.
I am using the same `detect` `build` and `release` script as [fabric-samples](https://github.com/hyperledger/fabric-samples/tree/main/asset-transfer-basic/chaincode-external/sampleBuilder/bin), and the connection.json is like
```
{
"address": "chaincode-org1.fabric.svc.cluster.local:3000",
"dial_timeout": "10s",
"tls_required": false
}
```
The chaincode install works fine but after installing I found that there is a connection.json under `/var/hyperledger/production/externalbuilder/builds/{{chaincodeID}}/release/chaincode/server/` but nothing in the json file. So the chaincode invoke will failed because the peer does not know the url of external chaincode service.
But if I manually add the content to the empty connection.json file, the invoking works fine.
I tried to debug the external builder by redirecting stdout to stderr in release script. The script do copy the `connection.json`(not empty) to /tmp/{{chaincodeID}}/release but for some reason it is not correctly copied back to /var/hyperledger/production/....
Any one has solution for this?
Thanks!
roclee (Mon, 04 Oct 2021 06:10:40 GMT):
Hello everyone.
I am deploying a fabric network to Kubernetes with external chaincode service. But I met a quite weird problem.
I am using the same `detect` `build` and `release` script as [fabric-samples](https://github.com/hyperledger/fabric-samples/tree/main/asset-transfer-basic/chaincode-external/sampleBuilder/bin), and the connection.json is like
```
{
"address": "chaincode-org1.fabric.svc.cluster.local:3000",
"dial_timeout": "10s",
"tls_required": false
}
```
The chaincode install works fine but after installing I found that there is a connection.json under `/var/hyperledger/production/externalbuilder/builds/{{chaincodeID}}/release/chaincode/server/` but nothing in the json file. So the chaincode invoke will failed because the peer does not know the url of external chaincode service.
But if I manually add the content to the empty connection.json file, the invoking works fine.
I tried to debug the external builder by redirecting stdout to stderr in release script. The script do copy the `connection.json`(not empty) to /tmp/{{chaincodeID}}/release but for some reason it is not correctly copied back to /var/hyperledger/production/....
Any one has solution for this?
Thanks!
I asked this question in #fabric-questions channel but I found that maybe this channel is more appropriate.
roclee (Mon, 04 Oct 2021 06:10:40 GMT):
Hello everyone.
I am deploying a fabric network to Kubernetes with external chaincode service. But I met a quite weird problem.
I am using the same `detect` `build` and `release` script as [fabric-samples](https://github.com/hyperledger/fabric-samples/tree/main/asset-transfer-basic/chaincode-external/sampleBuilder/bin), and the connection.json is like
```
{
"address": "chaincode-org1.fabric.svc.cluster.local:3000",
"dial_timeout": "10s",
"tls_required": false
}
```
The chaincode install works fine but after installing I found that there is a connection.json under `/var/hyperledger/production/externalbuilder/builds/{{chaincodeID}}/release/chaincode/server/` but nothing in the json file. So the chaincode invoke will failed because the peer does not know the url of external chaincode service.
But if I manually add the content to the empty connection.json file, the invoking works fine.
I tried to debug the external builder by redirecting stdout to stderr in release script. The script do copy the `connection.json`(not empty) to /tmp/{{chaincodeID}}/release but for some reason it is not correctly copied back to /var/hyperledger/production/....
Any one has solution for this?
Thanks!
I asked this question in #fabric-questions channel but I found that maybe this channel is more appropriate.
-----------------------------
update: Tried to use [fabric-ccs-builder](https://github.com/hyperledgendary/fabric-ccs-builder) instead of detect, build, release script and problem solved.
awa (Thu, 14 Oct 2021 08:09:07 GMT):
Hello everyone,
awa (Thu, 14 Oct 2021 08:09:07 GMT):
Hello everyone,
I asked this in #fabric-questions but it's directly linked to kubernetes:
https://chat.hyperledger.org/channel/fabric-questions?msg=sn4tTkZetZKdsj2fw
knagware9 (Mon, 18 Oct 2021 06:05:51 GMT):
Yes check this fabric samples https://github.com/hyperledger/fabric-samples/tree/main/test-network-k8s
nkaramolegos (Mon, 18 Oct 2021 09:35:10 GMT):
Thank you. I am already in that. It is very good work
rjones (Wed, 23 Mar 2022 17:35:20 GMT):
rjones (Wed, 23 Mar 2022 17:35:20 GMT):
rjones (Wed, 23 Mar 2022 17:35:20 GMT):
