Posted indocker 虚拟化

docker 容器上编译 go 程序无法运行，提示找不到文件

2018年3月19日

在kubernetes1.9上测试deploy的rolling-update，使用的测试用例是go写的一个简单webserver，本地编译没问题，且可以执行，但是再kubectl上跑起来提示文件找不到，网上搜了半天，起原因就是go的编译方式问题，因为运行在 alpine3.6 上面，涉及到了动态编译和静态编译的问题，网上的很多方法并不适用，最后测试了好久终于给搞定了。

初始编译语句如下：

GOOS=linux GOARCH=amd64 go build -o rollingupdate${TAG} main.go

执行如下命令的进行编译，生成docker镜像并push到私有仓库中
# make all
GOOS=linux GOARCH=amd64 go build -o rollingupdatev2 main.go
docker build -t repo.sudops.com/lib/rollingupdate:v2 .
Sending build context to Docker daemon  5.845MB
Step 1/4 : FROM alpine:3.6
 ---> 77144d8c6bdc
Step 2/4 : MAINTAINER sudops
 ---> Using cache
 ---> 9fcbe583a63c
Step 3/4 : ADD rollingupdatev2 /
 ---> d117e49eb4c2
Step 4/4 : ENTRYPOINT ["/rollingupdatev2"]
 ---> Running in 6b96e34edeca
Removing intermediate container 6b96e34edeca
 ---> b73af7d5ba9d
Successfully built b73af7d5ba9d
Successfully tagged repo.sudops.com/lib/rollingupdate:v2
docker push repo.sudops.com/lib/rollingupdate:v2
The push refers to repository [repo.sudops.com/lib/rollingupdate]
689d957bba62: Pushed
9dfa40a0da3b: Layer already exists
v2: digest: sha256:8a3324f536c7716d1204403f3358ee25710c294c2befae8f4f6f0381ff25a6ac size: 739

发现docker运行失败: 

# kubectl get pods
NAME                                   READY     STATUS             RESTARTS   AGE
rolling-update-test-6799b5d9d5-fml2k   0/1       CrashLoopBackOff   3          1m 
rolling-update-test-6799b5d9d5-jp9qb   0/1       CrashLoopBackOff   3          1m 
rolling-update-test-6799b5d9d5-wnfs6   0/1       CrashLoopBackOff   3          1m

先看看错误日志提示：
主要看这一行：panic: standard_init_linux.go:178: exec user process caused “no such file or directory” 

panic: standard_init_linux.go:178: exec user process caused "no such file or directory" [recovered]
	panic: standard_init_linux.go:178: exec user process caused "no such file or directory"
goroutine 1 [running, locked to thread]:
github.com/urfave/cli.HandleAction.func1(0xc4200a17a0)

k8s-master root@ffq16-vm04:~/yaml/rollingupdate/v1# kubectl logs rolling-update-test-6799b5d9d5-hrqqh -f
panic: standard_init_linux.go:178: exec user process caused "no such file or directory" [recovered]
	panic: standard_init_linux.go:178: exec user process caused "no such file or directory"
goroutine 1 [running, locked to thread]:
github.com/urfave/cli.HandleAction.func1(0xc4200af7a0)
	/builddir/build/BUILD/docker-3e8e77dcb88db0530c839b249bea7d75f9cd01d7/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x23f
panic(0x6f0cc0, 0xc420122a10)
	/usr/lib/golang/src/runtime/panic.go:489 +0x2cf
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0xc4200af208, 0xc42000e078, 0xc4200af2a8)
	/builddir/build/BUILD/docker-3e8e77dcb88db0530c839b249bea7d75f9cd01d7/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0xc1
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0xc420050640, 0xaa88e0, 0xc420122a10)
	/builddir/build/BUILD/docker-3e8e77dcb88db0530c839b249bea7d75f9cd01d7/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x353
main.glob..func8(0xc42007c780, 0x0, 0x0)
	/builddir/build/BUILD/docker-3e8e77dcb88db0530c839b249bea7d75f9cd01d7/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/main_unix.go:26 +0x66
reflect.Value.call(0x6d9be0, 0x750350, 0x13, 0x73b942, 0x4, 0xc4200af760, 0x1, 0x1, 0xc4200af6f0, 0x731420, ...)
	/usr/lib/golang/src/reflect/value.go:434 +0x91f
reflect.Value.Call(0x6d9be0, 0x750350, 0x13, 0xc4200af760, 0x1, 0x1, 0x665b36, 0x73bace, 0x4)
	/usr/lib/golang/src/reflect/value.go:302 +0xa4
github.com/urfave/cli.HandleAction(0x6d9be0, 0x750350, 0xc42007c780, 0x0, 0x0)
	/builddir/build/BUILD/docker-3e8e77dcb88db0530c839b249bea7d75f9cd01d7/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x18f
github.com/urfave/cli.Command.Run(0x73baee, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x74d391, 0x51, 0x0, ...)
	/builddir/build/BUILD/docker-3e8e77dcb88db0530c839b249bea7d75f9cd01d7/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xac8
github.com/urfave/cli.(*App).Run(0xc4200ca000, 0xc42000c140, 0x2, 0x2, 0x0, 0x0)
	/builddir/build/BUILD/docker-3e8e77dcb88db0530c839b249bea7d75f9cd01d7/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0x5d6
main.main()
	/builddir/build/BUILD/docker-3e8e77dcb88db0530c839b249bea7d75f9cd01d7/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/main.go:137 +0xbd2

根据网上的类似问题，尝试了如下两种都没有成功：
（1）编译时禁止cgo，CGO_ENABLED=0
（2）使用–ldflags “-extldflags -static” 来让gcc使用静态编译

最后找到在alpine3.6上应该使用网络编译go，即增加 -tags netgo。
编译改成

	GOOS=linux GOARCH=amd64 go build -tags netgo -o rollingupdate${TAG} main.go

再次运行 make all
发现这次docker运行成功。

编译后的文件： 

# file rollingupdate
hellov2: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped

# ldd rollingupdate
	not a dynamic executable

运行的pod如下：

# kubectl get pods
NAME                                  READY     STATUS    RESTARTS   AGE
rolling-update-test-8ddfb9c56-29vfc   1/1       Running   2          24m
rolling-update-test-8ddfb9c56-fggml   1/1       Running   0          24m
rolling-update-test-8ddfb9c56-x2n9r   1/1       Running   0          24m
Tags: