环境说明

因在日常维护过程中升级了docker的版本 ，碰到了一个 docker 的一个 bug, 导致 docker 服务无法正常启动。在恢复 docker 服务的过程中把 /var/lib/docker/containers 下的所有文件给 清空 了，清空后 docker 服务可以正常运行，但发现原来的服务中还存留着一个还在使用的 cmdb 系统也被我跟着删除了，是之前的前辈部署的。好在自己有 做备份 的习惯，现记录一下被删除容器的恢复过程。

  • 操作系统: Centos

  • Docker 版本: 19.03.8

  • 需要恢复容器: glpi & mysql

    一开始以为只要一个容器，后面恢复起来才发现还存在一个关联的外置数据库需要恢复。

容器恢复流程说明

目前我想到的恢复方案有两种:

  • 方案一：重建容器，基于数据卷的恢复，因为我们只删除了 /var/lib/docker/containers , 所有数据卷还是存在的。下面的步骤也将具体讲述使用这种方式进行恢复

  • 方案二：回滚， 将备份数据覆盖到 /var/lib/docker 下， 即:

    1
2
3
4
5
6
7

    cd /var/lib/docker.bak/ 

mv /var/lib/docker{,.bak-del}

cp -a /var/lib/docker.bak /var/lib/docker  # 备份为 /var/lib/docker.bak

service docker restart

    但是这里就回到了第一个问题上就是 docker 服务还是会存在无法启动。 docker 服务无法启动的原因及正确解决流程我会后面再后面写一篇博客说明。

实行方案一, 进行数据恢复

进入备份文件夹下，查找关键信息

遍历所有文件夹，查找和容器相关联的容器id

1
2

cd /var/lib/docker.bak/ 
for i in `ls */config.v2.json` ;do grep -l 'glpi' "$i";done # 遍历所有文件夹，查找和容器相关联的容器id

可以看到我们找到了，两个文件

image-20201228202756832

查看第一个配置文件，并使用 json 格式化工具 进行分析

1

cat 33c749acc76fec662a292e7a7fb73a1e32bd2b57ebe4270bb1ad46abc4598011/config.v2.json

image-20201228202917046

找到了容器运行的镜像名称即 fjudith/glpi

检查到容器部署时，使用的端口映射

image-20201228203739158

内部 80端口 映射到了 宿主机 的30000 端口

检查到 使用的存储卷

image-20201228203159841

使用了两个储存卷，一个为 宿主机的文件 ，一个为创建的的 glpi-app 存储卷

容器环境变量

image-20201228203903711

检查查找到的容器镜像是否匹配

1

docker image ls

image-20201228203018126

可以看到找到的镜像是 匹配 的

查看第二个配置文件，并使用 json 格式化工具 进行分析

1

cat a967e8f43115ea574ce1929e75b99374b4f050b612264928983baa0fd8109609/config.v2.json

image-20201228204649277

可以看到这是一个 数据库容器

查看 容器对应挂载的数据是否还在

image-20201228204913461

Dockerhub 查找相关容器信息

在 dockerhub 上 search fjudith/glpi 镜像，可以看到容器的启动命令。

image-20201228203520126

可以看到 dockerhub上 此容器的部署说明，和我们查看到了信息也吻合，那么确定了之前的前辈是一个什么部署步骤了

容器恢复

数据库容器恢复

数据库容器 具体 json 文件如下所示

格式化后

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180

{
    "StreamConfig":{

    },
    "State":{
        "Running":true,
        "Paused":false,
        "Restarting":false,
        "OOMKilled":false,
        "RemovalInProgress":false,
        "Dead":false,
        "Pid":8598,
        "ExitCode":0,
        "Error":"",
        "StartedAt":"2020-12-28T03:12:01.012923553Z",
        "FinishedAt":"2020-12-28T03:10:01.310689449Z",
        "Health":null
    },
    "ID":"a967e8f43115ea574ce1929e75b99374b4f050b612264928983baa0fd8109609",
    "Created":"2019-12-16T07:00:49.830939878Z",
    "Managed":false,
    "Path":"docker-entrypoint.sh",
    "Args":[
        "mysqld"
    ],
    "Config":{
        "Hostname":"a967e8f43115",
        "Domainname":"",
        "User":"",
        "AttachStdin":false,
        "AttachStdout":false,
        "AttachStderr":false,
        "ExposedPorts":{
            "3306/tcp":{

            }
        },
        "Tty":false,
        "OpenStdin":false,
        "StdinOnce":false,
        "Env":[
            "MYSQL_DATABASE=glpi",
            "MYSQL_ROOT_PASSWORD=pass",
            "MYSQL_USER=glpi",
            "MYSQL_PASSWORD=pass",
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "GOSU_VERSION=1.10",
            "GPG_KEYS=177F4010FE56CA3336300305F1656F24C74CD1D8",
            "MARIADB_MAJOR=10.4",
            "MARIADB_VERSION=1:10.4.11+maria~bionic"
        ],
        "Cmd":[
            "mysqld"
        ],
        "ArgsEscaped":true,
        "Image":"mariadb",
        "Volumes":{
            "/var/lib/mysql":{

            }
        },
        "WorkingDir":"",
        "Entrypoint":[
            "docker-entrypoint.sh"
        ],
        "OnBuild":null,
        "Labels":{

        }
    },
    "Image":"sha256:e93d99aa9076c3582e36fd458be51d2d389cf421b711f03b8767f156be4d2cfb",
    "NetworkSettings":{
        "Bridge":"",
        "SandboxID":"7e381858bd5aa515ccd13b8c856779132d6ddfcefd7ec82ece585db25c4a1c8c",
        "HairpinMode":false,
        "LinkLocalIPv6Address":"",
        "LinkLocalIPv6PrefixLen":0,
        "Networks":{
            "bridge":{
                "IPAMConfig":null,
                "Links":null,
                "Aliases":null,
                "NetworkID":"b34a6d28cfffb257837e9336d7556eb97ce768c835c59246361db5042c30394d",
                "EndpointID":"50432a9bac088866479de0e423926924d20494ad7e6054bb9395ff28572506bb",
                "Gateway":"172.17.0.1",
                "IPAddress":"172.17.0.2",
                "IPPrefixLen":16,
                "IPv6Gateway":"",
                "GlobalIPv6Address":"",
                "GlobalIPv6PrefixLen":0,
                "MacAddress":"02:42:ac:11:00:02",
                "IPAMOperational":false
            }
        },
        "Service":null,
        "Ports":{
            "3306/tcp":[
                {
                    "HostIp":"0.0.0.0",
                    "HostPort":"33306"
                }
            ]
        },
        "SandboxKey":"/var/run/docker/netns/7e381858bd5a",
        "SecondaryIPAddresses":null,
        "SecondaryIPv6Addresses":null,
        "IsAnonymousEndpoint":false,
        "HasSwarmEndpoint":false
    },
    "LogPath":"/var/lib/docker/containers/a967e8f43115ea574ce1929e75b99374b4f050b612264928983baa0fd8109609/a967e8f43115ea574ce1929e75b99374b4f050b612264928983baa0fd8109609-json.log",
    "Name":"/glpi-db",
    "Driver":"overlay",
    "MountLabel":"",
    "ProcessLabel":"",
    "RestartCount":0,
    "HasBeenStartedBefore":true,
    "HasBeenManuallyStopped":false,
    "MountPoints":{
        "/etc/localtime":{
            "Source":"/etc/localtime",
            "Destination":"/etc/localtime",
            "RW":true,
            "Name":"",
            "Driver":"",
            "Type":"bind",
            "Spec":{
                "Type":"bind",
                "Source":"/etc/localtime",
                "Target":"/etc/localtime"
            }
        },
        "/etc/mysql":{
            "Source":"/data/glpi-mysql/conf",
            "Destination":"/etc/mysql",
            "RW":true,
            "Name":"",
            "Driver":"",
            "Type":"bind",
            "Spec":{
                "Type":"bind",
                "Source":"/data/glpi-mysql/conf",
                "Target":"/etc/mysql"
            }
        },
        "/var/lib/mysql":{
            "Source":"/data/glpi-mysql/data",
            "Destination":"/var/lib/mysql",
            "RW":true,
            "Name":"",
            "Driver":"",
            "Type":"bind",
            "Spec":{
                "Type":"bind",
                "Source":"/data/glpi-mysql/data",
                "Target":"/var/lib/mysql"
            }
        },
        "/var/log/mysql":{
            "Source":"/data/glpi-mysql/log",
            "Destination":"/var/log/mysql",
            "RW":true,
            "Name":"",
            "Driver":"",
            "Type":"bind",
            "Spec":{
                "Type":"bind",
                "Source":"/data/glpi-mysql/log",
                "Target":"/var/log/mysql"
            }
        }
    },
    "SecretReferences":null,
    "AppArmorProfile":"",
    "HostnamePath":"/var/lib/docker/containers/a967e8f43115ea574ce1929e75b99374b4f050b612264928983baa0fd8109609/hostname",
    "HostsPath":"/var/lib/docker/containers/a967e8f43115ea574ce1929e75b99374b4f050b612264928983baa0fd8109609/hosts",
    "ShmPath":"/var/lib/docker/containers/a967e8f43115ea574ce1929e75b99374b4f050b612264928983baa0fd8109609/shm",
    "ResolvConfPath":"/var/lib/docker/containers/a967e8f43115ea574ce1929e75b99374b4f050b612264928983baa0fd8109609/resolv.conf",
    "SeccompProfile":"",
    "NoNewPrivileges":false
}

从 json 文件中推理出，容器启动命令，进行重建

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

docker run --name='glpi-db' -d \
--restart=always \
-p 33306:3306 \
-e MYSQL_DATABASE=glpi \
-e MYSQL_ROOT_PASSWORD=pass \
-e MYSQL_USER=glpi \
-e MYSQL_PASSWORD=pass \
-v /data/glpi-mysql/data:/var/lib/mysql \
-v /data/glpi-mysql/log:/var/log/mysql \
-v /data/glpi-mysql/conf:/etc/mysql \
-v /etc/localtime:/etc/localtime \
mariadb   # 启动

检查容器是否正常启动

1

docker logs --tail 100 -f glpi-db   # 检测启动日志

image-20201228210556103

使用数据库管理工具进行连接

image-20201228210726779

自此数据库 恢复成功

恢复 glpi 容器

glpi 容器具体 json 文件如下所示

格式后

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162

{
    "StreamConfig":{

    },
    "State":{
        "Running":true,
        "Paused":false,
        "Restarting":false,
        "OOMKilled":false,
        "RemovalInProgress":false,
        "Dead":false,
        "Pid":8894,
        "ExitCode":0,
        "Error":"",
        "StartedAt":"2020-12-28T03:12:03.393664003Z",
        "FinishedAt":"2020-12-28T03:09:59.174014953Z",
        "Health":null
    },
    "ID":"33c749acc76fec662a292e7a7fb73a1e32bd2b57ebe4270bb1ad46abc4598011",
    "Created":"2019-12-16T07:30:55.091572697Z",
    "Managed":false,
    "Path":"/docker-entrypoint.sh",
    "Args":[
        "apache2-foreground"
    ],
    "Config":{
        "Hostname":"33c749acc76f",
        "Domainname":"",
        "User":"",
        "AttachStdin":false,
        "AttachStdout":false,
        "AttachStderr":false,
        "ExposedPorts":{
            "80/tcp":{

            }
        },
        "Tty":false,
        "OpenStdin":false,
        "StdinOnce":false,
        "Env":[
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "PHPIZE_DEPS=autoconf 	dpkg-dev 	file 	g++ 	gcc 	libc-dev 	make 	pkg-config 	re2c",
            "PHP_INI_DIR=/usr/local/etc/php",
            "APACHE_CONFDIR=/etc/apache2",
            "APACHE_ENVVARS=/etc/apache2/envvars",
            "PHP_EXTRA_BUILD_DEPS=apache2-dev",
            "PHP_EXTRA_CONFIGURE_ARGS=--with-apxs2",
            "PHP_CFLAGS=-fstack-protector-strong -fpic -fpie -O2",
            "PHP_CPPFLAGS=-fstack-protector-strong -fpic -fpie -O2",
            "PHP_LDFLAGS=-Wl,-O1 -Wl,--hash-style=both -pie",
            "GPG_KEYS=1729F83938DA44E27BA0F4D3DBDB397470D12172 B1B44D8F021E4E2D6021E995DC9FF8D3EE5AF27F",
            "PHP_VERSION=7.2.4",
            "PHP_URL=https://secure.php.net/get/php-7.2.4.tar.xz/from/this/mirror",
            "PHP_ASC_URL=https://secure.php.net/get/php-7.2.4.tar.xz.asc/from/this/mirror",
            "PHP_SHA256=7916b1bd148ddfd46d7f8f9a517d4b09cd8a8ad9248734e7c8dd91ef17057a88",
            "PHP_MD5=",
            "GLPI_VERSION=9.2.1",
            "GLPI_URL=https://github.com/glpi-project/glpi/releases/download/9.2.1/glpi-9.2.1.tgz",
            "TERM=xterm"
        ],
        "Cmd":[
            "apache2-foreground"
        ],
        "ArgsEscaped":true,
        "Image":"fjudith/glpi",
        "Volumes":null,
        "WorkingDir":"/var/www/html",
        "Entrypoint":[
            "/docker-entrypoint.sh"
        ],
        "OnBuild":null,
        "Labels":{

        }
    },
    "Image":"sha256:4c740260749c175d94078f60dedcd898343d287268c9de8af539677e9f49acc9",
    "NetworkSettings":{
        "Bridge":"",
        "SandboxID":"30d50522da0bacc5d6813ecad985ae8891e68ebee54499c17ffabf546b7f6193",
        "HairpinMode":false,
        "LinkLocalIPv6Address":"",
        "LinkLocalIPv6PrefixLen":0,
        "Networks":{
            "bridge":{
                "IPAMConfig":null,
                "Links":null,
                "Aliases":null,
                "NetworkID":"b34a6d28cfffb257837e9336d7556eb97ce768c835c59246361db5042c30394d",
                "EndpointID":"b6cdb139c21fcf9f0c90283e83e52464d2d078af2f6ea48bcd764c562be5509a",
                "Gateway":"172.17.0.1",
                "IPAddress":"172.17.0.4",
                "IPPrefixLen":16,
                "IPv6Gateway":"",
                "GlobalIPv6Address":"",
                "GlobalIPv6PrefixLen":0,
                "MacAddress":"02:42:ac:11:00:04",
                "IPAMOperational":false
            }
        },
        "Service":null,
        "Ports":{
            "80/tcp":[
                {
                    "HostIp":"0.0.0.0",
                    "HostPort":"30000"
                }
            ]
        },
        "SandboxKey":"/var/run/docker/netns/30d50522da0b",
        "SecondaryIPAddresses":null,
        "SecondaryIPv6Addresses":null,
        "IsAnonymousEndpoint":false,
        "HasSwarmEndpoint":false
    },
    "LogPath":"/var/lib/docker/containers/33c749acc76fec662a292e7a7fb73a1e32bd2b57ebe4270bb1ad46abc4598011/33c749acc76fec662a292e7a7fb73a1e32bd2b57ebe4270bb1ad46abc4598011-json.log",
    "Name":"/glpi-app",
    "Driver":"overlay",
    "MountLabel":"",
    "ProcessLabel":"",
    "RestartCount":0,
    "HasBeenStartedBefore":true,
    "HasBeenManuallyStopped":false,
    "MountPoints":{
        "/etc/localtime":{
            "Source":"/etc/localtime",
            "Destination":"/etc/localtime",
            "RW":true,
            "Name":"",
            "Driver":"",
            "Type":"bind",
            "Spec":{
                "Type":"bind",
                "Source":"/etc/localtime",
                "Target":"/etc/localtime"
            }
        },
        "/var/www/html/files":{
            "Source":"/var/lib/docker/volumes/glpi-app/_data",
            "Destination":"/var/www/html/files",
            "RW":true,
            "Name":"glpi-app",
            "Driver":"local",
            "Type":"volume",
            "Relabel":"z",
            "ID":"f700b3bf921f366d7f29aa481757fb6dfd77090c940e4e7c2258b3690acc1168",
            "Spec":{
                "Type":"volume",
                "Source":"glpi-app",
                "Target":"/var/www/html/files"
            }
        }
    },
    "SecretReferences":null,
    "AppArmorProfile":"",
    "HostnamePath":"/var/lib/docker/containers/33c749acc76fec662a292e7a7fb73a1e32bd2b57ebe4270bb1ad46abc4598011/hostname",
    "HostsPath":"/var/lib/docker/containers/33c749acc76fec662a292e7a7fb73a1e32bd2b57ebe4270bb1ad46abc4598011/hosts",
    "ShmPath":"/var/lib/docker/containers/33c749acc76fec662a292e7a7fb73a1e32bd2b57ebe4270bb1ad46abc4598011/shm",
    "ResolvConfPath":"/var/lib/docker/containers/33c749acc76fec662a292e7a7fb73a1e32bd2b57ebe4270bb1ad46abc4598011/resolv.conf",
    "SeccompProfile":"",
    "NoNewPrivileges":false
}

从 json 文件中推理出，容器启动命令，进行重建

重建前，检查一下备份数据中，对应 docker 存储卷中的数据是否还在，可以看到文件什么的多还在 ，按道理我们只需要把文件 copy 到对应的文件夹下应该就可以了

image-20201228204135727

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

docker volume create "glpi-app"  # 创建 容器卷文件

cp -a /var/lib/docker.bak/volumes/glpi-app/_data/* /var/lib/docker/volumes/glpi-app/  # 将备份数据 copy 至新建的卷容器文件夹内

docker run --name=glpi-app -d \
--restart=always \
-p 30000:80 \
-v glpi-app:/var/www/html/files \
-v /etc/localtime:/etc/localtime \
--links glpi-db:mysql \
fjudith/glpi    # 尝试启动，需要注意网卡link的名称为之前启动的数据库容器的名称

image-20201228212927377

⚠️ 无法启动

-- links 在高版本的 Docker 中好像被移除了，文档说明

image-20201228212956200

使用替换 -- links 方案

创建容器

此方案将删除前面部署的 mysql 容器

1

docker rm -f glpi-db

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

docker network create glpi  # 创建网络

# 将前面部署的 "glpi-db", 更名为 "mysql" ，方便在同一个网络时使用

docker run --name=mysql -d \
--restart=always \
--network=glpi \
-p 33306:3306 \
-e MYSQL_DATABASE=glpi \
-e MYSQL_ROOT_PASSWORD=pass \
-e MYSQL_USER=glpi \
-e MYSQL_PASSWORD=pass \
-v /data/glpi-mysql/data:/var/lib/mysql \
-v /data/glpi-mysql/log:/var/log/mysql \
-v /data/glpi-mysql/conf:/etc/mysql \
-v /etc/localtime:/etc/localtime \
mariadb  


docker logs --tail 100 -f mysql  # 检查是否正常启动，正常后随后启动第二个容器

docker run --name=glpi-app -d \
--restart=always \
-p 30000:80 \
-v glpi-app:/var/www/html/files \
-v /etc/localtime:/etc/localtime \
--network=glpi \
fjudith/glpi

dashboard 引导恢复

选择 “更新” => “选择现有数据库”

image-20201228213751995

image-20201228214059990

登录账户后，数据回来了 😺