About neohope

一直在努力,还没想过要放弃...

InfluxDB环境搭建01

一、设计原则
InfluxDB是一个时间序列数据库,其性能优化方向为大规模、高效的时间序列数据的插入及查询操作。经权衡,部分场景下需要以牺牲功能为代价来提高性能,以下列出了部分设计理念:
1、数据按时间戳顺序存储,不允许重复数据存在。
2、数据删除是低频操作,批量数据删除除外;
3、数据更新是低频操作,实际场景很少发生;对某一数据的持续更新,不应该发生;绝大多数的数据都是新产生的数据,而且不会发生变化;
4、数据写入的时间戳接近当前时间,数据写入顺序与数据时间戳顺序一致;
5、数据规模至关重要,数据库性能优化,主要针对大规模的写入和读取操作,而不是更新和删除操作;
6、数据库可用性高于强一致性;
7、很多时间序列都是短暂,而不是永恒的产生数据;
8、不需要特别关注某个时刻的数据,从整体分析数据趋势才更重要;

二、基本概念
时间序列数据库,有一些特有的概念。其实只需要正确理解measurement、tag、series和存储文件之间的关系就可以比较容易的理解整个系统了。

概念 解释 对应于SQL
时间戳timestamp 在InfluxDB里的所有时间都是UTC的,时间戳timestamp就是一个UTC时刻下,某一序列series在该时刻的全部数据的唯一标识。知道了时间戳timestamp就可以获取序列series下指定时间的数据点point,这也意味着同一series下时间戳不允许重复。
同一个序列series中,两个相同时间戳数据的写入,会触发数据更新及数据合并;
类似于SQL中的自增主键,但是时间戳类型的
数据点point 一个序列series中,具有相同timestamp的的field数据集合。也就是说在同一个series中同一个timestamp的数据点point只会出现一次(可以视为在series中数据点以时间戳timestamp为主键存储)。
数据点可以保存一个measurement的field的一个或多个,也可以随时增加新的field。但对于已存储的数据,不允许修改数据类型。
类似于SQL中的一行记录
数据库database 一个包含schema、时序数据、数据保留策略、持续查询、用户、权限的逻辑集合。 类似于SQL中的数据库
模式schema schema包含database,retention policy,series,measurement,tag key,tag value以及field keys,其确定了在InfluxDB里面如何组织数据。 对应于SQL的数据库模式+表模式
度量measurement 用于描述数据如何保存的数据结构,包含了tag,field和时间序列数据。measurement是字符串。
一个度量measurement可以有不同的存续策略retention policy。
对用于SQL的表
序列series measurement中的逻辑的数据分组,也是实际的数据存储分组。由shared measurement,tag set和feild key组成,不包含field value。
换句话就是,在InfluxDB中,数据不是按measurement存的,而是按series存的。
类似于SQL中按某一列的值,进行分表操作
序列候选series cardinality 在InfluxDB实例中,由database、measurement、tag set、field key确定的不重复的组合数量。
举例说明一下:比如记录一个人员信息的measurement,只有两个tag(性别和是否已就业),其中性别为2个可选值(M、F),是否已就业为2个可选值(Y、N),那么这个measurement下会存在2×2共4个series cardinality。而实际存储时,数据也会按每个series进行处理。
类似于SQL中按某一列的值,可以分表的数量
field InfluxDB数据中记录metadata和数据的键值对。field数量必须大于0,不支持无索引。 对应于SQL的无索引列
field set 数据点上field key和field value的集合。 对应于SQL的无索引列名称和数值的集合
field key 组成field的键值对里面的键的部分。field key是字符串且保存在metadata中。 对应于SQL的无索引列的名称
field value 组成field的键值对里面的值的部分。field value才是真正的数据,可以是字符串,浮点数,整数,布尔型数据。
一个field value总是和一个timestamp相关联。
field value不会被索引,如果要对field value做过滤话,那就必须遍历所选时间范围里面的所有数据点,这种方式对比与tag效率会差很多。
对应于SQL的无索引列的值
tag InfluxDB数据中记录metadata的键值对。tags用于存储常用的metadata,在InfluxDB的数据中是可选的。tags会被索引,因此tag上的查询是很高效的。 对应于SQL的索引列
tag set 数据点上tag key和tag value的集合。 对应于SQL的无索引列名称和数值的集合
tag key 组成tag的键值对中的键部分,tag key是字符串,存在metadata中。 对应于SQL的无索引列名称
tag value 组成tag的键值对中的值部分,tag value是字符串,存在metadata中。 对应于SQL的无索引列数值
分片shard 在分布式部署的情况下,shard决定了数据要存几份以及存在哪里。shard中包含实际的编码和压缩数据,并由磁盘上的TSM文件表示。每个shard都属于唯一的一个shard group。一个shard group中可以有1到多个shard。每个shard包含一组特定的序列series。同一shard group中的同一series上的所有点将存储在磁盘上的相同shard(TSM文件)中。 和SQL不太一样,有些类似于kafka分布式存储partition的机制
分片存续期间shard duration shard duration决定了每个shard group跨越多少时间。具体间隔由retention policy中的SHARD DURATION决定。
例如,如果retention policy的SHARD DURATION设置为1w,则每个shard group将跨越一周,并包含时间戳在该周内的所有点。之前的数据将被删除。
和SQL不太一样,有些类似于kafka中日志保留时间log.retention.hours
分片组shard group shard group是shard的逻辑组合。shard group由时间和retention policy组织。
包含数据的retention policy至少包含一个关联的shard group。
一个shard group包含shard group覆盖的间隔的数据的所有shard。
每个shard group跨越的间隔是shard的持续时间。
这个和kafka分组就差别很大了
存续策略retention policy 描述了InfluxDB保存数据的长短(duration),数据存在集群里面的副本数(replication factor),以及shard group的时间范围(shard group duration)。
当新建database的时候,InfluxDB会自动创建一个叫做autogen的retention policy,其duration为永远,replication factor为1,shard group的duration设为的7天。
retention policy在每个database里面是唯一的,是series的重要组成部分。
TSM(Time Structured Merge tree) 一种InfluxDB的专用数据存储格式,比现有的B+或LSM树实现更大的压缩和更高的写入和读取吞吐量。 存储结果,类似于SQL中的B+树,或KV数据库中的LSM树。
每秒写入量values per second InfluxDB数据持久化速度的度量,等于将每秒写入的数据点数 乘以 每个数据点存储的值的数量。
例如,如果数据共有四个field,并且每秒处理10个batch,每个batch是5000个数据点。那么每秒10个batch x 每batch5000个数据点 x values per second是每数据点4个field =每秒200,000个值。
数据点缓存WAL(Write Ahead Log) 为了TSM数据文件访问频率,InfluxDB将最新的数据点缓存在WAL中,直到其总大小或时间达到系统阈值,会触发将数据以batch的方式写入到TSM文件中。
WAL中的数据点是可以被查询的。
而且WAL中的树节点已经完成了持久化,系统重启后仍然会被保留。InfluxDB进程启动时,会确保WAL中的所有点都写入到TSM中,然后才会接受新的数据写入,。
WAL与TSM的关系,和SQL日志以及B+树关系有些相似。
持续查询Continuous Query(CQ) Continuous Query(CQ)是在数据库内部,自动的周期性运行的一个InfluxQL的查询。
Continuous Query(CQ)需要在SELECT语句中使用一个函数,并且一定包括一个GROUP BY time()语句。
有些类似于SQL中的存储过程+JOB
桶bucket 在InfluxDB 2.0中,桶是一个有命名的,时间序列数据存储位置。
在InfluxDB 1.8+版本中, 一个database和一个retention policy的组合就代表了一个桶bucket。
2.0新增概念

三、数据存储对比
我们以官方文档中的foodships表为例子:
包含了3个带索引里的列park_id,planet,and time;和一个不带索引的列#_foodships。
在SQL里,foodships表看起来是这个样子的:

+---------+---------+---------------------+--------------+
| park_id | planet  | time                | #_foodships  |
+---------+---------+---------------------+--------------+
|       1 | Earth   | 1429185600000000000 |            0 |
|       1 | Earth   | 1429185601000000000 |            3 |
|       1 | Earth   | 1429185602000000000 |           15 |
|       1 | Earth   | 1429185603000000000 |           15 |
|       2 | Saturn  | 1429185600000000000 |            5 |
|       2 | Saturn  | 1429185601000000000 |            9 |
|       2 | Saturn  | 1429185602000000000 |           10 |
|       2 | Saturn  | 1429185603000000000 |           14 |
|       3 | Jupiter | 1429185600000000000 |           20 |
|       3 | Jupiter | 1429185601000000000 |           21 |
|       3 | Jupiter | 1429185602000000000 |           21 |
|       3 | Jupiter | 1429185603000000000 |           20 |
|       4 | Saturn  | 1429185600000000000 |            5 |
|       4 | Saturn  | 1429185601000000000 |            5 |
|       4 | Saturn  | 1429185602000000000 |            6 |
|       4 | Saturn  | 1429185603000000000 |            5 |
+---------+---------+---------------------+--------------+

但在InfluxDB中,要按索引列也就是tag划分series,每个series要按时间存储,看起来是这样的:

name: foodships
tags: park_id=1, planet=Earth
time			 #_foodships
----			 ------------
2015-04-16T12:00:00Z	 0
2015-04-16T12:00:01Z	 3
2015-04-16T12:00:02Z	 15
2015-04-16T12:00:03Z	 15

name: foodships
tags: park_id=2, planet=Saturn
time			 #_foodships
----			 ------------
2015-04-16T12:00:00Z	 5
2015-04-16T12:00:01Z	 9
2015-04-16T12:00:02Z	 10
2015-04-16T12:00:03Z	 14

name: foodships
tags: park_id=3, planet=Jupiter
time			 #_foodships
----			 ------------
2015-04-16T12:00:00Z	 20
2015-04-16T12:00:01Z	 21
2015-04-16T12:00:02Z	 21
2015-04-16T12:00:03Z	 20

name: foodships
tags: park_id=4, planet=Saturn
time			 #_foodships
----			 ------------
2015-04-16T12:00:00Z	 5
2015-04-16T12:00:01Z	 5
2015-04-16T12:00:02Z	 6
2015-04-16T12:00:03Z	 5

四、schema设计原则
1、建议将meta data做成tag
2、建议不要用数据关键字作为tag或field的名称
3、不建议使用过多的series
4、不建议tag和field采用相同名称
5、不建议用measurement的名称作为区分数据的依据
6、不建议在一个tag中塞入多个分类的信息

五、小结
基于以上原则和概念,我们对时间序列数据库有了一个初步的认识:
1、InfluxDB是一个无模式(schemaless)的数据库,你可以在任意时间添加measurement,tags和fields,但对于已存储的数据,是不允许修改数据类型的
2、对比与SQL数据库,InfluxDB其实更接近与KV数据库,而且InfluxDB对列扩展的支持是十分好的
3、和SQL不同,不支持跨measurement的JOIN
4、measurement和series的关系,是通过tag来划分的,这是和SQL十分不一样的地方
5、InfluxDB中数据分两大类tag和field,其中:tag是用来划分series用的,只能是数量可控的字符串,而且支持索引;field是真正用于数据存储的
6、由于优先考虑数据插入及数据查询的性能,而不是数据更新和数据删除,InfluxDB不能算一个完整的CRUD数据库,更像是一个CR-ud数据库:
更新某个数据点时,只需要在同一个时间戳下重复insert一次数据;
删除某个数据点时,是无法根据field的值进行删除操作的,遇到这种情况就需要先进行一次查询,然后通过时间戳进行删除;
不允许更新或重命名tag,遇到这种情况需要新建一个tag,迁移数据后,drop原tag的series;
不允许通过tag key删除tag,但可以直接drop整个series;
7、单条数据的更新和删除操作,尽量少做,效率比较低
8、时间戳及其重要,数据的组织是按时间戳进行的
9、支持数据分片、数据保存时间、数据保存份数的设置
10、十分适合按时间为维度进行采样、生产和更新的系统

CEPH环境搭建09

本节介绍如何编译ceph和nfs-ganesha源码:

1、ceph编译

git clone git://github.com/ceph/ceph
cd ceph
git checkout v15.2.3
git submodule update --init --recursive
./install-deps.sh
ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo" ./do_cmake.sh 
cd build
make -j4
make install

2、nfs-ganesha编译

#这一段移动到“CEPH环境搭建06”去了
#只需要将版本从V3.2换成V3.3即可

3、这两个编译后,需要手工安装FSAL库

cp nfs-ganesha/build/FSAL/FSAL_CEPH/libfsalceph.so /usr/lib/x86_64-linux-gnu/ganesha/
cp nfs-ganesha/build/FSAL/FSAL_RGW/libfsalrgw.so /usr/lib/x86_64-linux-gnu/ganesha/

完全从源码编译后,用GANESHA确实可以成功挂载RGW,但是空的,而且是只读。

CEPH环境搭建08

本节主要是将rbd(也就是块存储),通过LIO映射为iscsi(也就是ip san),从而提供给其他系统使用。这种方式比fuse后再映射出来性能要好,因为fuse是要经过内核的,要做切换。

0、原本计划使用容器继续处理,但没能成功,如果你成功了麻烦告诉我一下如何做的。

#这条路没测试成功,原理上应该是一样的
ceph orch daemon add iscsi rbd --placement="1 ceph-0002"

后面是通过编译源码的方式来完成的。

1、我们在“CEPH环境搭建03”中已经建立了rbd pool,而且建立了r1,r2,r3三个存储设备

rbd ls
r1
r2
r4

2、升级linux内核

#ceph-0002
#当前内核版本比较低,会导致LIO操作失败,表现为在所以必须升级
#在gwcli做attach或create硬盘时,会报错
#Issuing disk create/update request Failed : disk create/update failed on xxx. LUN allocation failure...
#去看rbd-target-api的日志,会发现:
#Could not set LIO device attribute cmd_time_out/qfull_time_out for device: rbd/r2. Kernel not supported. - error(Cannot find attribute: qfull_time_out)
uname -a
4.15.0-91-generic x86_64 GNU/Linux

# 查看可以使用的内核版本
apt list | grep linux-generic
linux-generic/bionic-updates,bionic-security 4.15.0.101.91 amd64 [upgradable from: 4.15.0.91.83]
linux-generic-hwe-16.04/bionic-updates,bionic-security 4.15.0.101.91 amd64
linux-generic-hwe-16.04-edge/bionic-updates,bionic-security 4.15.0.101.91 amd64
linux-generic-hwe-18.04/bionic-updates,bionic-security 5.3.0.53.110 amd64
linux-generic-hwe-18.04-edge/bionic-updates,bionic-security 5.3.0.53.110 amd64

# 安装高版本内核
apt-get install linux-generic-hwe-18.04-edge

# 重启
reboot

3、编译并发布程序

#ceph-0002
#安装所需包
apt install pkg-config libglib2.0-dev librbd1 libnl-3-200 libkmod2

#python2
pip install kmod pyudev urwid pyparsing rados rbd netifaces crypto requests  flask
#缺少两个包,暂时没有影响
#pip install gobject, python-openssl

#python3
apt-get install python3-pip python3-dev python3-openssl
apt install python-dev python3-pyparsing
pip3 install gobject  pyudev urwid pyparsing netifaces crypto requests  flask
#缺少三个包,暂时没有影响
#pip3 install kmod rados rbd

#tcmu-runner
git clone https://github.com/open-iscsi/tcmu-runner
#修改脚本,extra/install_dep.sh,在debianh后增加",ubuntu",让ubuntu与debian相同处理即可
./extra/install_dep.sh
cmake -Dwith-glfs=false -Dwith-qcow=false -DSUPPORT_SYSTEMD=ON -DCMAKE_INSTALL_PREFIX=/usr
make install

#rtslib-fb
git clone https://github.com/open-iscsi/rtslib-fb.git
cd rtslib-fb
python setup.py install

#configshell-fb
git clone https://github.com/open-iscsi/configshell-fb.git
cd configshell-fb
python setup.py install

#targetcli-fb
git clone https://github.com/open-iscsi/targetcli-fb.git
cd targetcli-fb
python setup.py install
mkdir /etc/target
mkdir /var/target

#ceph-iscsi
git clone https://github.com/ceph/ceph-iscsi.git
cd ceph-iscsi
python setup.py install --install-scripts=/usr/bin
cp usr/lib/systemd/system/rbd-target-gw.service /lib/systemd/system
cp usr/lib/systemd/system/rbd-target-api.service /lib/systemd/system

#启动服务
systemctl daemon-reload
systemctl enable tcmu-runner
systemctl start tcmu-runner
systemctl enable rbd-target-gw
systemctl start rbd-target-gw
systemctl enable rbd-target-api
systemctl start rbd-target-api

4、修改配置文件

#ceph-0002
vi /etc/ceph/iscsi-gateway.cfg
[config]
# Name of the Ceph storage cluster. A suitable Ceph configuration file allowing
# access to the Ceph storage cluster from the gateway node is required, if not
# colocated on an OSD node.
cluster_name = ceph
pool = rbd
# cluster_client_name = client.igw.ceph-0002
minimum_gateways = 1
gateway_ip_list = 192.168.1.102,192.168.1.103

# Place a copy of the ceph cluster's admin keyring in the gateway's /etc/ceph
# drectory and reference the filename here
gateway_keyring = ceph.client.admin.keyring

# API settings.
# The API supports a number of options that allow you to tailor it to your
# local environment. If you want to run the API under https, you will need to
# create cert/key files that are compatible for each iSCSI gateway node, that is
# not locked to a specific node. SSL cert and key files *must* be called
# 'iscsi-gateway.crt' and 'iscsi-gateway.key' and placed in the '/etc/ceph/' directory
# on *each* gateway node. With the SSL files in place, you can use 'api_secure = true'
# to switch to https mode.

# To support the API, the bear minimum settings are:
api_secure = false

# Additional API configuration options are as follows, defaults shown.
# api_user = admin
# api_password = admin
# api_port = 5001
# trusted_ip_list = 192.168.1.101,192.168.1.102,192.168.1.103,192.168.1.104

5、将节点加入监控

#ceph-0001
ceph dashboard iscsi-gateway-list
{"gateways": {}}

ceph dashboard iscsi-gateway-add http://admin:admin@ceph-0002:5000
Success

ceph dashboard iscsi-gateway-list
{"gateways": {"localhost.vm": {"service_url": "http://admin:admin@ceph-0002:5000"}}}

# 这个localhost.vm要记住

6、配置iscsi

#ceph-0004
gwcli

# 创建gateway
> /> cd /iscsi-target
> /iscsi-target...-igw/gateways> create localhost.vm 192.168.1.102 skipchecks=true
OS version/package checks have been bypassed
Adding gateway, sync'ing 0 disk(s) and 0 client(s)
ok

> /iscsi-target...-igw/gateways> ls
o- gateways ......................... [Up: 1/1, Portals: 1]
o- localhost.vm ..................... [192.168.1.102 (UP)]

# 挂载硬盘
> /iscsi-target...-igw/gateways> cd /disks
> /disks> attach rbd/r2
ok

> /disks> ls
o- disks ............................... [1G, Disks: 1]
o- rbd ..................................... [rbd (1G)]
o- r2 ................................... [rbd/r2 (1G)]
> /disks> attach rbd/r4
ok
> /disks> ls
o- disks ............................... [2G, Disks: 2]
o- rbd ..................................... [rbd (2G)]
o- r2 ................................... [rbd/r2 (1G)]
o- r4 ................................... [rbd/r4 (1G)]

#授权管理
> /iscsi-targets> cd iqn.2020-06.com.neohope.iscsi-gw:iscsi-igw/
> /iscsi-target...-gw:iscsi-igw> ls
o- iqn.2020-06.com.neohope.iscsi-gw:iscsi-igw ...[Auth: None, Gateways: 1]
o- disks ..................................... [Disks: 0]
o- gateways ....................... [Up: 1/1, Portals: 1]
| o- localhost.vm .................. [192.168.1.102 (UP)]
o- host-groups ............................. [Groups : 0]
o- hosts .................. [Auth: ACL_ENABLED, Hosts: 0]

# 创建initiator
> /iscsi-target...-gw:iscsi-igw> cd hosts
> /iscsi-target...csi-igw/hosts> create iqn.2020-06.com.neohope:ceph-0004
ok

> /iscsi-target...ope:ceph-0004> ls
o- iqn.2020-06.com.neohope:ceph-0004 ... [Auth: None, Disks: 0(0.00Y)]

# 创建授权
> /iscsi-target...ope:ceph-0004> auth username=myissicuid password=myissicpwdpwd
ok

# 分配硬盘
> /iscsi-target...ope:ceph-0004> disk add rbd/r2
ok

> /iscsi-target...ope:ceph-0004> ls
o- iqn.2020-06.com.neohope:ceph-0004 ......... [Auth: CHAP, Disks: 1(1G)]
o- lun 0 .............................. [rbd/r2(1G), Owner: localhost.vm]

7、挂载iscsi盘

#ceph-0004

#安装需要的软件
apt-get install open-iscsi

#查看可用的iscsi服务
iscsiadm -m discovery -t sendtargets -p 192.168.1.102
192.168.1.102:3260,1 iqn.2020-06.com.neohope.iscsi-gw:iscsi-igw

#修改配置文件,initiator要和上面分配的一致
vim /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2020-06.com.neohope:ceph-0004

#重启服务
systemctl status iscsid

#设置登录信息
iscsiadm -m node -T iqn.2020-06.com.neohope.iscsi-gw:iscsi-igw-o update --name node.session.auth.authmethod --value=CHAP
iscsiadm: No records found

iscsiadm -m node -T iqn.2020-06.com.neohope.iscsi-gw:iscsi-igw --op update --name node.session.auth.username --value=myissicuid

iscsiadm -m node -T iqn.2020-06.com.neohope.iscsi-gw:iscsi-igw --op update --name node.session.auth.password --value=myissicpwdpwd

#登录,挂载iscsi盘
iscsiadm -m node -T iqn.2020-06.com.neohope.iscsi-gw:iscsi-igw --login
Logging in to [iface: default, target: iqn.2020-06.com.neohope.iscsi-gw:iscsi-igw, portal: 192.168.1.102,3260] (multiple)
Login to [iface: default, target: iqn.2020-06.com.neohope.iscsi-gw:iscsi-igw, portal: 192.168.1.102,3260] successful.

8、使用iscsi盘

#ceph-0004

#查看硬盘,会发现多出一块
fdisk -l
Disk /dev/vda: 40 GiB, 42949672960 bytes, 83886080 sectors
Disk /dev/vdb: 20 GiB, 21474836480 bytes, 41943040 sectors
Disk /dev/mapper/ceph--44634c9f--cf41--4215--bd5b--c2db93659bf1-osd--block--b192f8e5--55f2--4e75--a7ce--54d007410829: 20 GiB, 21470642176 bytes, 41934848 sectors
Disk /dev/sda: 1 GiB, 1073741824 bytes, 2097152 sectors

#查看sda硬盘情况
fdisk -l /dev/sda
Disk /dev/sda: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
root@ceph-0004:/dev# fdisk -l /dev/sda
Disk /dev/sda: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

#格式化
sudo mkfs.ext4 -m0 /dev/sda
mke2fs 1.44.1 (24-Mar-2018)
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 42229c39-e23c-46b2-929d-469e66196498
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

#挂载
mount -t ext4 /dev/sda /mnt/iscsi

#基本操作
cd /mnt/iscsi/
ls
vi iscis.txt
ls

9、卸载iscsi盘

#ceph-0004

#取消mount
umount /mnt/iscsi

#登出
iscsiadm -m node -T iqn.2020-06.com.neohope.iscsi-gw:iscsi-igw  --logout
Logging out of session [sid: 1, target: iqn.2020-06.com.neohope.iscsi-gw:iscsi-igw, portal: 192.168.1.102,3260]
Logout of [sid: 1, target: iqn.2020-06.com.neohope.iscsi-gw:iscsi-igw, portal: 192.168.1.102,3260] successful.

#查看硬盘列表,会发现iscsi盘已经不见了
fdisk -l

CEPH环境搭建07

本节主要是将rbd(也就是块存储),通过tgt映射为iscsi(也就是ip san),从而提供给其他系统使用。

1、安装需要的软件

#ceph-0002
apt-get install tgt
apt-get install open-iscsi

2、查看tgt对rbd的支持情况

#ceph-0002
tgtadm --lld iscsi --op show --mode system
System:
  State: ready
  debug: off
LLDs:
  iscsi: ready
  iser: error
  Backing stores:
  sheepdog
  bsg
  sg
  null
  ssc
  smc (bsoflags sync:direct)
  mmc (bsoflags sync:direct)
  rdwr (bsoflags sync:direct)
  aio
Device types:
  disk
  cd/dvd
  osd
  controller
  changer
  tape
  passthrough
iSNS:
  iSNS=Off
  iSNSServerIP=
  iSNSServerPort=3205
  iSNSAccessControl=Off

可见当前版本的tgt不支持直接使用rbd,所以要先将rbd映射为硬盘后,再进行处理。

3、创建并映射rbd设备

#ceph-0002
#创建块存储
rbd create --size 1024 rbd/r2

#禁用特性
rbd feature disable r2 object-map fast-diff deep-flatten

#映射r2设备
rbd map r2
/dev/rbd0

#查看映射情况
rbd showmapped
id pool image snap device
0  rbd  r2    -    /dev/rbd0

4、修改tgt配置文件

#ceph-0002
vim /etc/tgt/targets.conf
<target iqn.2020-06.com.neohope:iscsi="">
  backing-store  /dev/rbd0             #虚拟设备
  initiator-address 192.168.1.0/24     #IP限制,请根据实际需要配置
  # incominguser iuid ipwd             #授权限制,请根据实际需要配置
  write-cache off                      #关闭缓存,请根据实际需要配置
</target>

# 重启服务,配置生效
systemctl restart tgt.service

# 查看tgt情况
tgt-admin –-show
Target 1: iqn.2020-06.com.neohope:iscsi
System information:
  Driver: iscsi
  State: ready
I_T nexus information:
LUN information:
  LUN: 0
    Type: controller
    SCSI ID: IET     00010000
    SCSI SN: beaf10
    Size: 0 MB, Block size: 1
    Online: Yes
    Removable media: No
    Prevent removal: No
    Readonly: No
    SWP: No
    Thin-provisioning: No
    Backing store type: null
    Backing store path: None
    Backing store flags:
  LUN: 1
    Type: disk
    SCSI ID: IET     00010001
    SCSI SN: beaf11
    Size: 1074 MB, Block size: 512
    Online: Yes
    Removable media: No
    Prevent removal: No
    Readonly: No
    SWP: No
    Thin-provisioning: No
    Backing store type: rdwr
    Backing store path: /dev/rbd0
    Backing store flags:
Account information:
ACL information:
  192.168.1.0/24

5、挂载iscsi盘

#ceph-0004
#安装需要的软件
apt-get install open-iscsi

#查看可用的iscsi服务
iscsiadm -m discovery -t sendtargets -p 192.168.1.102:3260
192.168.1.102:3260,1 iqn.2020-06.com.neohope:iscsi

#登录,挂载iscsi盘
iscsiadm -m node -T iqn.2020-06.com.neohope:iscsi --login
Logging in to [iface: default, target: iqn.2020-06.com.neohope:iscsi, portal: 192.168.1.102,3260] (multiple)
Login to [iface: default, target: iqn.2020-06.com.neohope:iscsi, portal: 192.168.1.102,3260] successful.

6、使用iscsi盘

#ceph-0004

#查看硬盘,会发现多出一块
fdisk -l
Disk /dev/vda: 40 GiB, 42949672960 bytes, 83886080 sectors
Disk /dev/vdb: 20 GiB, 21474836480 bytes, 41943040 sectors
Disk /dev/mapper/ceph--44634c9f--cf41--4215--bd5b--c2db93659bf1-osd--block--b192f8e5--55f2--4e75--a7ce--54d007410829: 20 GiB, 21470642176 bytes, 41934848 sectors
Disk /dev/sda: 1 GiB, 1073741824 bytes, 2097152 sectors

#查看sda硬盘情况
fdisk -l /dev/sda
Disk /dev/sda: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
root@ceph-0004:/dev# fdisk -l /dev/sda
Disk /dev/sda: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

#格式化
sudo mkfs.ext4 -m0 /dev/sda
mke2fs 1.44.1 (24-Mar-2018)
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 42229c39-e23c-46b2-929d-469e66196498
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

#挂载
mount -t ext4 /dev/sda /mnt/iscsi

#基本操作
cd /mnt/iscsi/
ls
vi iscis.txt
ls

7、卸载iscsi盘

#取消mount
umount /mnt/iscsi

#登出
iscsiadm -m node -T iqn.2020-06.com.neohope:iscsi  --logout
Logging out of session [sid: 1, target: iqn.2020-06.com.neohope:iscsi, portal: 192.168.1.102,3260]
Logout of [sid: 1, target: iqn.2020-06.com.neohope:iscsi, portal: 192.168.1.102,3260] successful.

#查看硬盘列表,会发现iscsi盘已经不见了
fdisk -l

CEPH环境搭建06

本节主要是使用ganesha将cephfs和rgw,映射为nfs,从而提供给其他系统使用。

0、原本计划使用容器继续处理,但没能成功,如果你成功了麻烦告诉我一下如何做的。

#这条路没测试成功,原理上应该是一样的
ceph orch apply nfs mynfs nfspool --placement="1 ceph-0003"

后面是通过编译源码的方式来完成的,操作系统是Ubuntu18.0.4,由于没有高版本的librgw的开发库(Ubuntu18.0.4只有1.1.6,而ganesha却要1.1.7以上),所以最后没能支持rgw,只能支持cephfs:

1、安装所需包

#ceph-0003
apt-get install gcc git cmake autoconf libtool bison flex
apt-get install libssl-dev  libnfs-dev doxygen libgssglue-dev libkrb5-dev liburcu-dev libntirpc-dev
apt-get install libnfsidmap-dev uuid-dev libblkid-dev
apt-get install librados-dev librgw-dev libcephfs-dev nfs-ganesha-ceph
snap install cmake  --classic
#缺少libzardfs_client,webclient,暂时没有影响

2、下载源码

#ceph-0003
git clone https://github.com/nfs-ganesha/nfs-ganesha.git
cd nfs-ganesha
git checkout V3.2
git submodule update --init --recursive

3、编译

#ceph-0003
mkdir build
cd build

#这里要把rgw的自持去掉,否则无法通过
ccmake ../src
>config
>generate

make
make install

4、创建ganesha配置文件

# ceph-0003
# 配置文件是支持cephfs和rgw的
vi /etc/ganesha/ganesha.conf
NFS_CORE_PARAM {
        Enable_NLM = false;
        Enable_RQUOTA = false;
        Protocols = 4;
        NFS_Port = 2049;
}

NFSv4 {
        Delegations = false;
        Minor_Versions = 1, 2;
}

CACHEINODE {
        Dir_Chunk = 0;
        NParts = 1;
        Cache_Size = 1;
}

EXPORT_DEFAULTS {
        Attr_Expiration_Time = 0;
}

EXPORT {
    Export_ID = 1001;
    Protocols = 3,4;
    Path = /;
    Pseudo = /cephfs;
    Access_Type = RW;
    #Transport_Protocols = TCP;
    Squash = Root_squash;
    FSAL {
        Name = CEPH;
    }
}

EXPORT
{
    Export_ID=1002;
    NFS_Protocols = 3,4;
    Path = "/";
    Pseudo = /rgw;
    Access_Type = RW;
    #Transport_Protocols = TCP;
    Squash = Root_squash;

    FSAL {
        Name = RGW;
        User_Id = "s3user";
        Access_Key_Id ="J5IWQUN50UJHVH48461Q";
        Secret_Access_Key = "5fgLtqd07EI6LS3PamPVNBpqwPPfPYJNxULBlgpj";
    }
}

LOG {
    Facility {
        name = FILE;
        destination = "/var/log/ganesha/ganesha.log";
        enable = active;
    }
}

CEPH {
    ceph_conf = '/etc/ceph/ceph.conf';
    cluster = "ceph";
    name = "rgw.myrealm.myzone";
}

RGW {
    ceph_conf = "/etc/ceph/ceph.conf";
}

5、运行ganesha

ganesha.nfsd -f /etc/ganesha/ganesha.conf -L /var/log/ganesha/ganesha.log -N NIV_INFO

6、挂载

#ceph-0004
mount -t nfs4 ceph-0003:/cephfs  /mnt/nfs/

ls /mnt/nfs
fuse.txt  volumes

cat /mnt/nfs/fuse.txt
fuse

ls /mnt/nfs/volumes/
subvg01  subvg02

7、取消挂载

#ceph-0004
umount /mnt/nfs

8、补充:
前面傻了,只用了ubuntu的源,忘记使用ceph的源了。其实用ceph的源后,就可以解决问题了,操作和上面相同,编译方式如下:

#添加ceph源
wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -
sudo apt-add-repository 'deb https://download.ceph.com/debian-octopus/ bionic main'
sudo apt-get update

#安装需要的软件
apt-get install gcc git cmake autoconf libtool bison flex
apt-get install libssl-dev  libnfs-dev doxygen libgssglue-dev libkrb5-dev liburcu-dev libntirpc-dev
apt-get install libnfsidmap-dev uuid-dev libblkid-dev
apt-get install librados-dev librgw-dev libcephfs-dev nfs-ganesha-ceph
snap install cmake  --classic
#缺少libzardfs_client,webclient,暂时没有影响

#下载源码并编译
git clone https://github.com/nfs-ganesha/nfs-ganesha.git
git checkout V3.2
git submodule update --init --recursive

mkdir build
cd build
ccmake ../src

make
make install

CEPH环境搭建05

本节主要是使用nfs-kernel-server将cephfs或rbd映射为nfs。

1、安装nfs相关软件

apt-get install nfs-kernel-server
apt-get install nfs-common

2、在ceph-0001挂载好fuse和rbd

ceph-fuse /mnt/fuse
rbd map r1
mount -t ext4 /dev/rbd0 /mnt/rbd

3、配置exports文件

vi /etc/exports
/mnt/fuse         192.168.1.0/24(rw,sync,no_root_squash,no_subtree_check,fsid=0)
/mnt/rbd          192.168.1.0/24(rw,sync,no_root_squash,no_subtree_check)

4、更新配置

exportfs -a

systemctl restart nfs-kernel-server

showmount -e
Export list for ceph-0001:
/mnt/rbd  192.168.1.0/24
/mnt/fuse 192.168.1.0/24

5、到另一台机器进行挂载

mount -t nfs  ceph-0001:/mnt/fuse  /mnt/fuse --verbose
mount.nfs: timeout set for Fri Jun  5 19:06:27 2020
mount.nfs: trying text-based options 'vers=4.2,addr=192.168.1.101,clientaddr=192.168.1.103'
mount.nfs: mount(2): No such file or directory
mount.nfs: trying text-based options 'addr=192.168.1.101'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.1.101 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.1.101 prog 100005 vers 3 prot UDP port 39630

mount -t nfs  ceph-0001:/mnt/rbd  /mnt/rbd --verbose
mount.nfs: timeout set for Fri Jun  5 19:06:36 2020
mount.nfs: trying text-based options 'vers=4.2,addr=192.168.1.101,clientaddr=192.168.1.103'
mount.nfs: mount(2): No such file or directory
mount.nfs: trying text-based options 'addr=192.168.1.101'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.1.101 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.1.101 prog 100005 vers 3 prot UDP port 39630

ls /mnt/fuse
fuse.txt  volumes

ls /mnt/rbd
lost+found  rbd1.txt

CEPH环境搭建04

本节主要是测试ceph的三种存储方式之一对象存储。

1、创建zone,并开启rgw

#创建realm
radosgw-admin realm create --rgw-realm=myrealm --default
#创建zonegroup
radosgw-admin zonegroup create --rgw-zonegroup=myzg --endpoints=http://ceph01:8080 --rgw-realm=myrealm --master --default
#创建zone
radosgw-admin zone create --rgw-zonegroup=myzg --rgw-zone=myzone --endpoints=http://ceph01:8080 --master --default
# 在ceph01启用rgw
ceph orch apply rgw myrealm myzone --placement="1 ceph01"

2、创建s3用户

radosgw-admin user create --uid=s3user --display-name=s3user  --system
{
  "user_id": "s3user",
  "display_name": "s3user",
  "email": "",
  "suspended": 0,
  "max_buckets": 1000,
  "auid": 0,
  "subusers": [],
  "keys": [
    {
      "user": "s3user",
      "access_key": "6IUA1DMFDTP5BG9ZMIR8",
      "secret_key": "zdoRS2yWL6EsNEBa4xuOSFMPn0lMvPJVMIYZJirP"
    }
  ],
  "swift_keys": [],
  "caps": [],
  "op_mask": "read, write, delete",
  "system": "true",
  "default_placement": "",
  "placement_tags": [],
  "bucket_quota": {
    "enabled": false,
    "check_on_raw": false,
    "max_size": -1,
    "max_size_kb": 0,
    "max_objects": -1
  },
  "user_quota": {
    "enabled": false,
    "check_on_raw": false,
    "max_size": -1,
    "max_size_kb": 0,
    "max_objects": -1
  },
  "temp_url_keys": [],
  "type": "rgw"
}

3、创建swift用户

sudo radosgw-admin subuser create --uid=s3user --subuser=s3user:swift --access=full
{
  "user_id": "s3user",
  "display_name": "s3user",
  "email": "",
  "suspended": 0,
  "max_buckets": 1000,
  "auid": 0,
  "subusers": [
    {
      "id": "s3user:swift",
      "permissions": "full-control"
    }
  ],
  "keys": [
    {
      "user": "s3user",
      "access_key": "6IUA1DMFDTP5BG9ZMIR8",
      "secret_key": "zdoRS2yWL6EsNEBa4xuOSFMPn0lMvPJVMIYZJirP"
    }
  ],
  "swift_keys": [
    {
      "user": "s3user:swift",
      "secret_key": "2wou5DxQ6WiBYyHf8qb3QIMX9BnhhBd5Njlj6LJX"
    }
  ],
  "caps": [],
  "op_mask": "read, write, delete",
  "system": "true",
  "default_placement": "",
  "placement_tags": [],
  "bucket_quota": {
    "enabled": false,
    "check_on_raw": false,
    "max_size": -1,
    "max_size_kb": 0,
    "max_objects": -1
  },
  "user_quota": {
    "enabled": false,
    "check_on_raw": false,
    "max_size": -1,
    "max_size_kb": 0,
    "max_objects": -1
  },
  "temp_url_keys": [],
  "type": "rgw"
}

4、通过s3接口新增一个bucket

sudo apt-get install python-boto

# 编辑s3test.py文件
vi s3test.py

import boto.s3.connection

access_key = '6IUA1DMFDTP5BG9ZMIR8'
secret_key = 'zdoRS2yWL6EsNEBa4xuOSFMPn0lMvPJVMIYZJirP'

conn = boto.connect_s3(
  aws_access_key_id=access_key,
  aws_secret_access_key=secret_key,
  host='ceph01', port=80,
  is_secure=False, calling_format=boto.s3.connection.OrdinaryCallingFormat(),
  )

bucket = conn.create_bucket('my-new-bucket')
for bucket in conn.get_all_buckets():
  print "{name} {created}".format(
    name=bucket.name,
    created=bucket.creation_date,
    )

#运行
python s3test.py
mybucket 2020-05-19T20:01:59.139Z

5、通过swift接口查询bucket

sudo apt-get install python-pip
sudo pip install --upgrade setuptools
sudo pip install --upgrade python-swiftclient

# 用swift工具查询bucket
swift -V 1 -A http://172.16.172.101:80/auth -U s3user:swift -K '2wou5DxQ6WiBYyHf8qb3QIMX9BnhhBd5Njlj6LJX' list
mybucket

5、通过s3cmd接口查询bucket

sudo apt-get install s3cmd

#生成配置
s3cmd --configure

#用s3cmd工具查询bucket
s3cmd ls
2020-05-19 20:01  s3://mybucket

CEPH环境搭建03

本节主要是测试ceph的三种存储方式之一块存储。

1、创建存储池以及rbd

sudo ceph osd pool ls
sudo ceph osd pool create rbd

sudo rados df

sudo rbd ls
sudo rbd create --size 1024 rbd/r1

2、将映像映射到块设备

#直接映射会报一个错
#sudo rbd map r1
#rbd: sysfs write failed
#RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable r1 object-map fast-diff deep-flatten".
#In some cases useful info is found in syslog - try "dmesg | tail".
#rbd: map failed: (6) No such device or address

#修正错误,重新映射
sudo rbd feature disable r1 object-map fast-diff deep-flatten
sudo rbd map r1
/dev/rbd0

#查看映射情况
sudo rbd showmapped
id pool image snap device
0  rbd  r1    -    /dev/rbd0

3、初始化块设备

#fdis初始化设备
sudo fdisk -l /dev/rbd0

#格式化分区为ext4
sudo mkfs.ext4 -m0 /dev/rbd0

#挂载块设备
sudo mkdir -p /mnt/rbd/r1
sudo mount -t ext4 /dev/rbd0 /mnt/rbd/r1

4、进行一些基本操作

sudo ls /mnt/rbd/r1

sudo vi /mnt/rbd/r1/hi.txt

sudo cat /mnt/rbd/r1/hi.txt
hello rbd

CEPH环境搭建02

本节主要是将存储设备加入ceph进行管理,并测试ceph的三种存储方式之一cephfs。

1、查看设备状态

sudo ceph osd status
ID  HOST     USED  AVAIL  WR OPS  WR DATA  RD OPS  RD DATA  STATE
0  ceph01  1027M   298G      0        0       0        0   exists,up
1  ceph02  1027M   298G      0        0       0        0   exists,up
2  ceph03  1027M   298G      0        0       0        0   exists,up
3  ceph04  1027M   298G      0        0       0        0   exists,up

sudo ceph orch device ls
HOST    PATH      TYPE   SIZE  DEVICE                             AVAIL  REJECT REASONS
ceph01  /dev/sdb  hdd    300G  VBOX_HARDDISK_VB434b1565-528a303a  True
ceph01  /dev/sda  hdd    300G  VBOX_HARDDISK_VB3eec2162-9aed4ffc  False  locked
ceph02  /dev/sdb  hdd    300G  VBOX_HARDDISK_VBa6445865-c497aa8e  True
ceph02  /dev/sda  hdd    300G  VBOX_HARDDISK_VB64e04201-60c7209f  False  locked
ceph03  /dev/sdb  hdd    300G  VBOX_HARDDISK_VB20fd0c04-b14ef3fa  True
ceph03  /dev/sda  hdd    300G  VBOX_HARDDISK_VB6f4439ab-85f80c78  False  locked
ceph04  /dev/sdb  hdd    300G  VBOX_HARDDISK_VB2c293541-3183e992  True
ceph04  /dev/sda  hdd    300G  VBOX_HARDDISK_VBd81d45d4-a88d6ff3  False  locked

2、存储设备加入osd管理

sudo ceph orch apply osd --all-available-devices
Scheduled osd update...

sudo ceph orch device ls --refresh
HOST    PATH      TYPE   SIZE  DEVICE                             AVAIL  REJECT REASONS
ceph01  /dev/sda  hdd    300G  VBOX_HARDDISK_VB3eec2162-9aed4ffc  False  locked
ceph01  /dev/sdb  hdd    300G  VBOX HARDDISK_VB434b1565-528a303a  False  LVM detected, Insufficient space (&amp;amp;lt;5GB) on vgs, locked
ceph02  /dev/sda  hdd    300G  VBOX_HARDDISK_VB64e04201-60c7209f  False  locked
ceph02  /dev/sdb  hdd    300G  VBOX HARDDISK_VBa6445865-c497aa8e  False  LVM detected, locked, Insufficient space (&amp;amp;lt;5GB) on vgs
ceph03  /dev/sda  hdd    300G  VBOX_HARDDISK_VB6f4439ab-85f80c78  False  locked
ceph03  /dev/sdb  hdd    300G  VBOX HARDDISK_VB20fd0c04-b14ef3fa  False  locked, Insufficient space (&amp;amp;lt;5GB) on vgs, LVM detected
ceph04  /dev/sda  hdd    300G  VBOX_HARDDISK_VBd81d45d4-a88d6ff3  False  locked
ceph04  /dev/sdb  hdd    300G  VBOX HARDDISK_VB2c293541-3183e992  False  LVM detected, locked, Insufficient space (&amp;amp;lt;5GB) on vgs

3、查看osd状态

sudo ceph osd df
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP  META   AVAIL    %USE  VAR   PGS  STATUS
0    hdd  0.29300   1.00000  300 GiB  1.0 GiB  3.4 MiB   0 B  1 GiB  299 GiB  0.33  1.00    0      up
1    hdd  0.29300   1.00000  300 GiB  1.0 GiB  3.4 MiB   0 B  1 GiB  299 GiB  0.33  1.00    1      up
2    hdd  0.29300   1.00000  300 GiB  1.0 GiB  3.4 MiB   0 B  1 GiB  299 GiB  0.33  1.00    1      up
3    hdd  0.29300   1.00000  300 GiB  1.0 GiB  3.4 MiB   0 B  1 GiB  299 GiB  0.33  1.00    1      up
TOTAL  1.2 TiB  4.0 GiB   14 MiB   0 B  4 GiB  1.2 TiB  0.33
MIN/MAX VAR: 1.00/1.00  STDDEV: 0

sudo ceph osd utilization
avg 0.75
stddev 0.433013 (expected baseline 0.75)
min osd.0 with 0 pgs (0 * mean)
max osd.1 with 1 pgs (1.33333 * mean)
neohope@ceph01:~$ sudo ceph osd pool stats
pool device_health_metrics id 1
nothing is going on

sudo ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME        STATUS  REWEIGHT  PRI-AFF
-1         1.17200  root default
-3         0.29300      host ceph01
0    hdd  0.29300          osd.0        up   1.00000  1.00000
-5         0.29300      host ceph02
1    hdd  0.29300          osd.1        up   1.00000  1.00000
-7         0.29300      host ceph03
2    hdd  0.29300          osd.2        up   1.00000  1.00000
-9         0.29300      host ceph04
3    hdd  0.29300          osd.3        up   1.00000  1.00000

sudo ceph pg stat
1 pgs: 1 active+clean; 0 B data, 14 MiB used, 1.2 TiB / 1.2 TiB avail

4、创建cephfs

sudo ceph fs volume ls
[]
sudo ceph fs volume create  v1
sudo ceph fs volume ls
[
{
"name": "v1"
}
]

sudo ceph fs subvolumegroup create v1 g1
sudo ceph fs subvolumegroup ls v1

sudo ceph fs subvolume create v1 sv1
sudo ceph fs subvolume ls v1

sudo ceph fs ls
name: v1, metadata pool: cephfs.v1.meta, data pools: [cephfs.v1.data ]

5、挂载cephfs

sudo apt-get install ceph-fuse

# 挂载cephfs
sudo mkdir -p /mnt/ceph/ceph_fuse
sudo ceph-fuse /mnt/ceph/ceph_fuse
ceph-fuse[24512]: starting ceph client
2020-05-18 05:57:36.039818 7f7d221a2500 -1 init, newargv = 0x559708e0e2e0 newarg                                                                                                             c=9
ceph-fuse[24512]: starting fuse

# 查看挂载情况
sudo mount | grep ceph
ceph-fuse on /mnt/ceph/ceph_fuse type fuse.ceph-fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

# 可以当作本地磁盘做一些日常操作
sudo ls /mnt/ceph/ceph_fuse/
volumes
sudo ls /mnt/ceph/ceph_fuse/volumes
g1  _nogroup
sudo ls /mnt/ceph/ceph_fuse/volumes/g1

sudo vi /mnt/ceph/ceph_fuse/volumes/g1/hi.txt
sudo cat /mnt/ceph/ceph_fuse/volumes/g1/hi.txt
hello ceph fuse

CEPH环境搭建01

1、初始环境

准备四个节点(每个节点的hosts和hostname要进行相应修改):

ceph-0001 172.16.172.101
ceph-0002 172.16.172.102
ceph-0003 172.16.172.103
ceph-0004 172.16.172.104

每个节点都执行:

sudo apt-get update
sudo apt-get install docker.io

2、在主节点安装cephadm

#官方推荐的方法有问题
#sudo ./cephadm add-repo --release octopus
#The key(s) in the keyring /etc/apt/trusted.gpg.d/ceph.release.gpg are ignored as the file has an unsupported filetype.
#sudo rm /etc/apt/trusted.gpg.d/ceph.release.gpg

wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -
echo deb http://download.ceph.com/debian-octopus/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
sudo apt-get update

curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm
chmod 711 cephadm
sudo ./cephadm install

sudo cephadm install ceph-common

3、初始化

sudo mkdir -p /etc/ceph

sudo cephadm bootstrap --mon-ip 172.16.172.101 --allow-overwrite
INFO:cephadm:Verifying podman|docker is present...
INFO:cephadm:Verifying lvm2 is present...
INFO:cephadm:Verifying time synchronization is in place...
INFO:cephadm:Unit systemd-timesyncd.service is enabled and running
INFO:cephadm:Repeating the final host check...
INFO:cephadm:podman|docker (/usr/bin/docker) is present
INFO:cephadm:systemctl is present
INFO:cephadm:lvcreate is present
INFO:cephadm:Unit systemd-timesyncd.service is enabled and running
INFO:cephadm:Host looks OK
INFO:root:Cluster fsid: 7bffaaf6-9688-11ea-ac24-080027b4217f
INFO:cephadm:Verifying IP 172.16.172.101 port 3300 ...
INFO:cephadm:Verifying IP 172.16.172.101 port 6789 ...
INFO:cephadm:Mon IP 172.16.172.101 is in CIDR network 172.16.172.0/24
INFO:cephadm:Pulling latest docker.io/ceph/ceph:v15 container...
INFO:cephadm:Extracting ceph user uid/gid from container image...
INFO:cephadm:Creating initial keys...
INFO:cephadm:Creating initial monmap...
INFO:cephadm:Creating mon...
INFO:cephadm:Waiting for mon to start...
INFO:cephadm:Waiting for mon...
INFO:cephadm:Assimilating anything we can from ceph.conf...
INFO:cephadm:Generating new minimal ceph.conf...
INFO:cephadm:Restarting the monitor...
INFO:cephadm:Setting mon public_network...
INFO:cephadm:Creating mgr...
INFO:cephadm:Wrote keyring to /etc/ceph/ceph.client.admin.keyring
INFO:cephadm:Wrote config to /etc/ceph/ceph.conf
INFO:cephadm:Waiting for mgr to start...
INFO:cephadm:Waiting for mgr...
INFO:cephadm:mgr not available, waiting (1/10)...
INFO:cephadm:mgr not available, waiting (2/10)...
INFO:cephadm:mgr not available, waiting (3/10)...
INFO:cephadm:Enabling cephadm module...
INFO:cephadm:Waiting for the mgr to restart...
INFO:cephadm:Waiting for Mgr epoch 5...
INFO:cephadm:Setting orchestrator backend to cephadm...
INFO:cephadm:Generating ssh key...
INFO:cephadm:Wrote public SSH key to to /etc/ceph/ceph.pub
INFO:cephadm:Adding key to root@localhost's authorized_keys...
INFO:cephadm:Adding host ceph01...
INFO:cephadm:Deploying mon service with default placement...
INFO:cephadm:Deploying mgr service with default placement...
INFO:cephadm:Deploying crash service with default placement...
INFO:cephadm:Enabling mgr prometheus module...
INFO:cephadm:Deploying prometheus service with default placement...
INFO:cephadm:Deploying grafana service with default placement...
INFO:cephadm:Deploying node-exporter service with default placement...
INFO:cephadm:Deploying alertmanager service with default placement...
INFO:cephadm:Enabling the dashboard module...
INFO:cephadm:Waiting for the mgr to restart...
INFO:cephadm:Waiting for Mgr epoch 13...
INFO:cephadm:Generating a dashboard self-signed certificate...
INFO:cephadm:Creating initial admin user...
INFO:cephadm:Fetching dashboard port number...
INFO:cephadm:Ceph Dashboard is now available at:

URL: https://localhost:8443/
User: admin
Password: mdbewc14gq

INFO:cephadm:You can access the Ceph CLI with:

sudo /usr/sbin/cephadm shell --fsid 7bffaaf6-9688-11ea-ac24-080027b4217f -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring

INFO:cephadm:Please consider enabling telemetry to help improve Ceph:

ceph telemetry on

For more information see:

https://docs.ceph.com/docs/master/mgr/telemetry/

INFO:cephadm:Bootstrap complete.

4、此时可以通过最后给出的信息,登录管理页面了

5、修改配置文件

sudo vi /etc/ceph/ceph.conf

[global]
fsid = a4547d9d-f1a1-4753-b5cc-df0e043ebc65
mon_initial_members = ceph01
#原本生成的mon_host好像有些问题
#mon_host = [v2:ceph01:3300/0,v1:ceph:6789/0]
mon_host = 172.16.172.101
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
public_network = 172.16.172.0/24

6、查看ceph状态

sudo ceph status
cluster:
id:     7bffaaf6-9688-11ea-ac24-080027b4217f
health: HEALTH_WARN
Reduced data availability: 1 pg inactive
OSD count 0 &amp;amp;amp;amp;amp;lt; osd_pool_default_size 3

services:
mon: 1 daemons, quorum ceph01 (age 35m)
mgr: ceph01.lreqdw(active, since 33m)
osd: 0 osds: 0 up, 0 in

data:
pools:   1 pools, 1 pgs
objects: 0 objects, 0 B
usage:   0 B used, 0 B / 0 B avail
pgs:     100.000% pgs unknown
1 unknown

7、三个子节点做好准备

#在ceph01
#将ceph.pub拷贝到其他三个节点
scp /etc/ceph/ceph.pub  neohope@ceph02:~/authorized_keys

#在ceph02
#启用root
sudo passwd -u root
#设置好root的authorized_keys
mv authorized_keys  /root/.ssh/
cd /root/.ssh/
chown root:root authorized_keys
chmod 0600 authorized_keys
#允许root ssh登录
sudo sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
sudo service ssh restart

#在ceph01
#获取私钥
sudo ceph config-key get mgr/cephadm/ssh_identity_key &amp;amp;amp;amp;amp;gt; ceph.pem
chmod 0600 ceph.pem
#测试root登录
ssh  -i ceph.pem root@ceph02

8、三个子节点加入网络

sudo ceph orch host add ceph02
Added host 'ceph02'

sudo ceph orch host add ceph03
Added host 'ceph03'

sudo ceph orch host add ceph04
Added host 'ceph04'

sudo ceph orch host ls
HOST    ADDR    LABELS  STATUS
ceph01  ceph01
ceph02  ceph02
ceph03  ceph03
ceph04  ceph04

9、设置mon

ceph orch apply mon 4
ceph orch apply mon ceph01,ceph02,ceph03,ceph04
sudo ceph status