CEPH环境搭建06

本节主要是使用ganesha将cephfs和rgw,映射为nfs,从而提供给其他系统使用。

0、原本计划使用容器继续处理,但没能成功,如果你成功了麻烦告诉我一下如何做的。

#这条路没测试成功,原理上应该是一样的
ceph orch apply nfs mynfs nfspool --placement="1 ceph-0003"

后面是通过编译源码的方式来完成的,操作系统是Ubuntu18.0.4,由于没有高版本的librgw的开发库(Ubuntu18.0.4只有1.1.6,而ganesha却要1.1.7以上),所以最后没能支持rgw,只能支持cephfs:

1、安装所需包

#ceph-0003
apt-get install gcc git cmake autoconf libtool bison flex
apt-get install libssl-dev  libnfs-dev doxygen libgssglue-dev libkrb5-dev liburcu-dev libntirpc-dev
apt-get install libnfsidmap-dev uuid-dev libblkid-dev
apt-get install librados-dev librgw-dev libcephfs-dev nfs-ganesha-ceph
snap install cmake  --classic
#缺少libzardfs_client,webclient,暂时没有影响

2、下载源码

#ceph-0003
git clone https://github.com/nfs-ganesha/nfs-ganesha.git
cd nfs-ganesha
git checkout V3.2
git submodule update --init --recursive

3、编译

#ceph-0003
mkdir build
cd build

#这里要把rgw的自持去掉,否则无法通过
ccmake ../src
>config
>generate

make
make install

4、创建ganesha配置文件

# ceph-0003
# 配置文件是支持cephfs和rgw的
vi /etc/ganesha/ganesha.conf
NFS_CORE_PARAM {
        Enable_NLM = false;
        Enable_RQUOTA = false;
        Protocols = 4;
        NFS_Port = 2049;
}

NFSv4 {
        Delegations = false;
        Minor_Versions = 1, 2;
}

CACHEINODE {
        Dir_Chunk = 0;
        NParts = 1;
        Cache_Size = 1;
}

EXPORT_DEFAULTS {
        Attr_Expiration_Time = 0;
}

EXPORT {
    Export_ID = 1001;
    Protocols = 3,4;
    Path = /;
    Pseudo = /cephfs;
    Access_Type = RW;
    #Transport_Protocols = TCP;
    Squash = Root_squash;
    FSAL {
        Name = CEPH;
    }
}

EXPORT
{
    Export_ID=1002;
    NFS_Protocols = 3,4;
    Path = "/";
    Pseudo = /rgw;
    Access_Type = RW;
    #Transport_Protocols = TCP;
    Squash = Root_squash;

    FSAL {
        Name = RGW;
        User_Id = "s3user";
        Access_Key_Id ="J5IWQUN50UJHVH48461Q";
        Secret_Access_Key = "5fgLtqd07EI6LS3PamPVNBpqwPPfPYJNxULBlgpj";
    }
}

LOG {
    Facility {
        name = FILE;
        destination = "/var/log/ganesha/ganesha.log";
        enable = active;
    }
}

CEPH {
    ceph_conf = '/etc/ceph/ceph.conf';
    cluster = "ceph";
    name = "rgw.myrealm.myzone";
}

RGW {
    ceph_conf = "/etc/ceph/ceph.conf";
}

5、运行ganesha

ganesha.nfsd -f /etc/ganesha/ganesha.conf -L /var/log/ganesha/ganesha.log -N NIV_INFO

6、挂载

#ceph-0004
mount -t nfs4 ceph-0003:/cephfs  /mnt/nfs/

ls /mnt/nfs
fuse.txt  volumes

cat /mnt/nfs/fuse.txt
fuse

ls /mnt/nfs/volumes/
subvg01  subvg02

7、取消挂载

#ceph-0004
umount /mnt/nfs

8、补充:
前面傻了,只用了ubuntu的源,忘记使用ceph的源了。其实用ceph的源后,就可以解决问题了,操作和上面相同,编译方式如下:

#添加ceph源
wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -
sudo apt-add-repository 'deb https://download.ceph.com/debian-octopus/ bionic main'
sudo apt-get update

#安装需要的软件
apt-get install gcc git cmake autoconf libtool bison flex
apt-get install libssl-dev  libnfs-dev doxygen libgssglue-dev libkrb5-dev liburcu-dev libntirpc-dev
apt-get install libnfsidmap-dev uuid-dev libblkid-dev
apt-get install librados-dev librgw-dev libcephfs-dev nfs-ganesha-ceph
snap install cmake  --classic
#缺少libzardfs_client,webclient,暂时没有影响

#下载源码并编译
git clone https://github.com/nfs-ganesha/nfs-ganesha.git
git checkout V3.2
git submodule update --init --recursive

mkdir build
cd build
ccmake ../src

make
make install

CEPH环境搭建05

本节主要是使用nfs-kernel-server将cephfs或rbd映射为nfs。

1、安装nfs相关软件

apt-get install nfs-kernel-server
apt-get install nfs-common

2、在ceph-0001挂载好fuse和rbd

ceph-fuse /mnt/fuse
rbd map r1
mount -t ext4 /dev/rbd0 /mnt/rbd

3、配置exports文件

vi /etc/exports
/mnt/fuse         192.168.1.0/24(rw,sync,no_root_squash,no_subtree_check,fsid=0)
/mnt/rbd          192.168.1.0/24(rw,sync,no_root_squash,no_subtree_check)

4、更新配置

exportfs -a

systemctl restart nfs-kernel-server

showmount -e
Export list for ceph-0001:
/mnt/rbd  192.168.1.0/24
/mnt/fuse 192.168.1.0/24

5、到另一台机器进行挂载

mount -t nfs  ceph-0001:/mnt/fuse  /mnt/fuse --verbose
mount.nfs: timeout set for Fri Jun  5 19:06:27 2020
mount.nfs: trying text-based options 'vers=4.2,addr=192.168.1.101,clientaddr=192.168.1.103'
mount.nfs: mount(2): No such file or directory
mount.nfs: trying text-based options 'addr=192.168.1.101'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.1.101 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.1.101 prog 100005 vers 3 prot UDP port 39630

mount -t nfs  ceph-0001:/mnt/rbd  /mnt/rbd --verbose
mount.nfs: timeout set for Fri Jun  5 19:06:36 2020
mount.nfs: trying text-based options 'vers=4.2,addr=192.168.1.101,clientaddr=192.168.1.103'
mount.nfs: mount(2): No such file or directory
mount.nfs: trying text-based options 'addr=192.168.1.101'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.1.101 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.1.101 prog 100005 vers 3 prot UDP port 39630

ls /mnt/fuse
fuse.txt  volumes

ls /mnt/rbd
lost+found  rbd1.txt

CEPH环境搭建04

本节主要是测试ceph的三种存储方式之一对象存储。

1、创建zone,并开启rgw

#创建realm
radosgw-admin realm create --rgw-realm=myrealm --default
#创建zonegroup
radosgw-admin zonegroup create --rgw-zonegroup=myzg --endpoints=http://ceph01:8080 --rgw-realm=myrealm --master --default
#创建zone
radosgw-admin zone create --rgw-zonegroup=myzg --rgw-zone=myzone --endpoints=http://ceph01:8080 --master --default
# 在ceph01启用rgw
ceph orch apply rgw myrealm myzone --placement="1 ceph01"

2、创建s3用户

radosgw-admin user create --uid=s3user --display-name=s3user  --system
{
  "user_id": "s3user",
  "display_name": "s3user",
  "email": "",
  "suspended": 0,
  "max_buckets": 1000,
  "auid": 0,
  "subusers": [],
  "keys": [
    {
      "user": "s3user",
      "access_key": "6IUA1DMFDTP5BG9ZMIR8",
      "secret_key": "zdoRS2yWL6EsNEBa4xuOSFMPn0lMvPJVMIYZJirP"
    }
  ],
  "swift_keys": [],
  "caps": [],
  "op_mask": "read, write, delete",
  "system": "true",
  "default_placement": "",
  "placement_tags": [],
  "bucket_quota": {
    "enabled": false,
    "check_on_raw": false,
    "max_size": -1,
    "max_size_kb": 0,
    "max_objects": -1
  },
  "user_quota": {
    "enabled": false,
    "check_on_raw": false,
    "max_size": -1,
    "max_size_kb": 0,
    "max_objects": -1
  },
  "temp_url_keys": [],
  "type": "rgw"
}

3、创建swift用户

sudo radosgw-admin subuser create --uid=s3user --subuser=s3user:swift --access=full
{
  "user_id": "s3user",
  "display_name": "s3user",
  "email": "",
  "suspended": 0,
  "max_buckets": 1000,
  "auid": 0,
  "subusers": [
    {
      "id": "s3user:swift",
      "permissions": "full-control"
    }
  ],
  "keys": [
    {
      "user": "s3user",
      "access_key": "6IUA1DMFDTP5BG9ZMIR8",
      "secret_key": "zdoRS2yWL6EsNEBa4xuOSFMPn0lMvPJVMIYZJirP"
    }
  ],
  "swift_keys": [
    {
      "user": "s3user:swift",
      "secret_key": "2wou5DxQ6WiBYyHf8qb3QIMX9BnhhBd5Njlj6LJX"
    }
  ],
  "caps": [],
  "op_mask": "read, write, delete",
  "system": "true",
  "default_placement": "",
  "placement_tags": [],
  "bucket_quota": {
    "enabled": false,
    "check_on_raw": false,
    "max_size": -1,
    "max_size_kb": 0,
    "max_objects": -1
  },
  "user_quota": {
    "enabled": false,
    "check_on_raw": false,
    "max_size": -1,
    "max_size_kb": 0,
    "max_objects": -1
  },
  "temp_url_keys": [],
  "type": "rgw"
}

4、通过s3接口新增一个bucket

sudo apt-get install python-boto

# 编辑s3test.py文件
vi s3test.py

import boto.s3.connection

access_key = '6IUA1DMFDTP5BG9ZMIR8'
secret_key = 'zdoRS2yWL6EsNEBa4xuOSFMPn0lMvPJVMIYZJirP'

conn = boto.connect_s3(
  aws_access_key_id=access_key,
  aws_secret_access_key=secret_key,
  host='ceph01', port=80,
  is_secure=False, calling_format=boto.s3.connection.OrdinaryCallingFormat(),
  )

bucket = conn.create_bucket('my-new-bucket')
for bucket in conn.get_all_buckets():
  print "{name} {created}".format(
    name=bucket.name,
    created=bucket.creation_date,
    )

#运行
python s3test.py
mybucket 2020-05-19T20:01:59.139Z

5、通过swift接口查询bucket

sudo apt-get install python-pip
sudo pip install --upgrade setuptools
sudo pip install --upgrade python-swiftclient

# 用swift工具查询bucket
swift -V 1 -A http://172.16.172.101:80/auth -U s3user:swift -K '2wou5DxQ6WiBYyHf8qb3QIMX9BnhhBd5Njlj6LJX' list
mybucket

5、通过s3cmd接口查询bucket

sudo apt-get install s3cmd

#生成配置
s3cmd --configure

#用s3cmd工具查询bucket
s3cmd ls
2020-05-19 20:01  s3://mybucket

CEPH环境搭建03

本节主要是测试ceph的三种存储方式之一块存储。

1、创建存储池以及rbd

sudo ceph osd pool ls
sudo ceph osd pool create rbd

sudo rados df

sudo rbd ls
sudo rbd create --size 1024 rbd/r1

2、将映像映射到块设备

#直接映射会报一个错
#sudo rbd map r1
#rbd: sysfs write failed
#RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable r1 object-map fast-diff deep-flatten".
#In some cases useful info is found in syslog - try "dmesg | tail".
#rbd: map failed: (6) No such device or address

#修正错误,重新映射
sudo rbd feature disable r1 object-map fast-diff deep-flatten
sudo rbd map r1
/dev/rbd0

#查看映射情况
sudo rbd showmapped
id pool image snap device
0  rbd  r1    -    /dev/rbd0

3、初始化块设备

#fdis初始化设备
sudo fdisk -l /dev/rbd0

#格式化分区为ext4
sudo mkfs.ext4 -m0 /dev/rbd0

#挂载块设备
sudo mkdir -p /mnt/rbd/r1
sudo mount -t ext4 /dev/rbd0 /mnt/rbd/r1

4、进行一些基本操作

sudo ls /mnt/rbd/r1

sudo vi /mnt/rbd/r1/hi.txt

sudo cat /mnt/rbd/r1/hi.txt
hello rbd

CEPH环境搭建02

本节主要是将存储设备加入ceph进行管理,并测试ceph的三种存储方式之一cephfs。

1、查看设备状态

sudo ceph osd status
ID  HOST     USED  AVAIL  WR OPS  WR DATA  RD OPS  RD DATA  STATE
0  ceph01  1027M   298G      0        0       0        0   exists,up
1  ceph02  1027M   298G      0        0       0        0   exists,up
2  ceph03  1027M   298G      0        0       0        0   exists,up
3  ceph04  1027M   298G      0        0       0        0   exists,up

sudo ceph orch device ls
HOST    PATH      TYPE   SIZE  DEVICE                             AVAIL  REJECT REASONS
ceph01  /dev/sdb  hdd    300G  VBOX_HARDDISK_VB434b1565-528a303a  True
ceph01  /dev/sda  hdd    300G  VBOX_HARDDISK_VB3eec2162-9aed4ffc  False  locked
ceph02  /dev/sdb  hdd    300G  VBOX_HARDDISK_VBa6445865-c497aa8e  True
ceph02  /dev/sda  hdd    300G  VBOX_HARDDISK_VB64e04201-60c7209f  False  locked
ceph03  /dev/sdb  hdd    300G  VBOX_HARDDISK_VB20fd0c04-b14ef3fa  True
ceph03  /dev/sda  hdd    300G  VBOX_HARDDISK_VB6f4439ab-85f80c78  False  locked
ceph04  /dev/sdb  hdd    300G  VBOX_HARDDISK_VB2c293541-3183e992  True
ceph04  /dev/sda  hdd    300G  VBOX_HARDDISK_VBd81d45d4-a88d6ff3  False  locked

2、存储设备加入osd管理

sudo ceph orch apply osd --all-available-devices
Scheduled osd update...

sudo ceph orch device ls --refresh
HOST    PATH      TYPE   SIZE  DEVICE                             AVAIL  REJECT REASONS
ceph01  /dev/sda  hdd    300G  VBOX_HARDDISK_VB3eec2162-9aed4ffc  False  locked
ceph01  /dev/sdb  hdd    300G  VBOX HARDDISK_VB434b1565-528a303a  False  LVM detected, Insufficient space (<5GB) on vgs, locked
ceph02  /dev/sda  hdd    300G  VBOX_HARDDISK_VB64e04201-60c7209f  False  locked
ceph02  /dev/sdb  hdd    300G  VBOX HARDDISK_VBa6445865-c497aa8e  False  LVM detected, locked, Insufficient space (<5GB) on vgs
ceph03  /dev/sda  hdd    300G  VBOX_HARDDISK_VB6f4439ab-85f80c78  False  locked
ceph03  /dev/sdb  hdd    300G  VBOX HARDDISK_VB20fd0c04-b14ef3fa  False  locked, Insufficient space (<5GB) on vgs, LVM detected
ceph04  /dev/sda  hdd    300G  VBOX_HARDDISK_VBd81d45d4-a88d6ff3  False  locked
ceph04  /dev/sdb  hdd    300G  VBOX HARDDISK_VB2c293541-3183e992  False  LVM detected, locked, Insufficient space (<5GB) on vgs

3、查看osd状态

sudo ceph osd df
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP  META   AVAIL    %USE  VAR   PGS  STATUS
0    hdd  0.29300   1.00000  300 GiB  1.0 GiB  3.4 MiB   0 B  1 GiB  299 GiB  0.33  1.00    0      up
1    hdd  0.29300   1.00000  300 GiB  1.0 GiB  3.4 MiB   0 B  1 GiB  299 GiB  0.33  1.00    1      up
2    hdd  0.29300   1.00000  300 GiB  1.0 GiB  3.4 MiB   0 B  1 GiB  299 GiB  0.33  1.00    1      up
3    hdd  0.29300   1.00000  300 GiB  1.0 GiB  3.4 MiB   0 B  1 GiB  299 GiB  0.33  1.00    1      up
TOTAL  1.2 TiB  4.0 GiB   14 MiB   0 B  4 GiB  1.2 TiB  0.33
MIN/MAX VAR: 1.00/1.00  STDDEV: 0

sudo ceph osd utilization
avg 0.75
stddev 0.433013 (expected baseline 0.75)
min osd.0 with 0 pgs (0 * mean)
max osd.1 with 1 pgs (1.33333 * mean)
neohope@ceph01:~$ sudo ceph osd pool stats
pool device_health_metrics id 1
nothing is going on

sudo ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME        STATUS  REWEIGHT  PRI-AFF
-1         1.17200  root default
-3         0.29300      host ceph01
0    hdd  0.29300          osd.0        up   1.00000  1.00000
-5         0.29300      host ceph02
1    hdd  0.29300          osd.1        up   1.00000  1.00000
-7         0.29300      host ceph03
2    hdd  0.29300          osd.2        up   1.00000  1.00000
-9         0.29300      host ceph04
3    hdd  0.29300          osd.3        up   1.00000  1.00000

sudo ceph pg stat
1 pgs: 1 active+clean; 0 B data, 14 MiB used, 1.2 TiB / 1.2 TiB avail

4、创建cephfs

sudo ceph fs volume ls
[]
sudo ceph fs volume create  v1
sudo ceph fs volume ls
[
{
"name": "v1"
}
]

sudo ceph fs subvolumegroup create v1 g1
sudo ceph fs subvolumegroup ls v1

sudo ceph fs subvolume create v1 sv1
sudo ceph fs subvolume ls v1

sudo ceph fs ls
name: v1, metadata pool: cephfs.v1.meta, data pools: [cephfs.v1.data ]

5、挂载cephfs

sudo apt-get install ceph-fuse

# 挂载cephfs
sudo mkdir -p /mnt/ceph/ceph_fuse
sudo ceph-fuse /mnt/ceph/ceph_fuse
ceph-fuse[24512]: starting ceph client
2020-05-18 05:57:36.039818 7f7d221a2500 -1 init, newargv = 0x559708e0e2e0 newarg                                                                                                             c=9
ceph-fuse[24512]: starting fuse

# 查看挂载情况
sudo mount | grep ceph
ceph-fuse on /mnt/ceph/ceph_fuse type fuse.ceph-fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

# 可以当作本地磁盘做一些日常操作
sudo ls /mnt/ceph/ceph_fuse/
volumes
sudo ls /mnt/ceph/ceph_fuse/volumes
g1  _nogroup
sudo ls /mnt/ceph/ceph_fuse/volumes/g1

sudo vi /mnt/ceph/ceph_fuse/volumes/g1/hi.txt
sudo cat /mnt/ceph/ceph_fuse/volumes/g1/hi.txt
hello ceph fuse

CEPH环境搭建01

1、初始环境

准备四个节点(每个节点的hosts和hostname要进行相应修改):

ceph-0001 172.16.172.101
ceph-0002 172.16.172.102
ceph-0003 172.16.172.103
ceph-0004 172.16.172.104

每个节点都执行:

sudo apt-get update
sudo apt-get install docker.io

2、在主节点安装cephadm

#官方推荐的方法有问题
#sudo ./cephadm add-repo --release octopus
#The key(s) in the keyring /etc/apt/trusted.gpg.d/ceph.release.gpg are ignored as the file has an unsupported filetype.
#sudo rm /etc/apt/trusted.gpg.d/ceph.release.gpg

wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -
echo deb http://download.ceph.com/debian-octopus/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
sudo apt-get update

curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm
chmod 711 cephadm
sudo ./cephadm install

sudo cephadm install ceph-common

3、初始化

sudo mkdir -p /etc/ceph

sudo cephadm bootstrap --mon-ip 172.16.172.101 --allow-overwrite
INFO:cephadm:Verifying podman|docker is present...
INFO:cephadm:Verifying lvm2 is present...
INFO:cephadm:Verifying time synchronization is in place...
INFO:cephadm:Unit systemd-timesyncd.service is enabled and running
INFO:cephadm:Repeating the final host check...
INFO:cephadm:podman|docker (/usr/bin/docker) is present
INFO:cephadm:systemctl is present
INFO:cephadm:lvcreate is present
INFO:cephadm:Unit systemd-timesyncd.service is enabled and running
INFO:cephadm:Host looks OK
INFO:root:Cluster fsid: 7bffaaf6-9688-11ea-ac24-080027b4217f
INFO:cephadm:Verifying IP 172.16.172.101 port 3300 ...
INFO:cephadm:Verifying IP 172.16.172.101 port 6789 ...
INFO:cephadm:Mon IP 172.16.172.101 is in CIDR network 172.16.172.0/24
INFO:cephadm:Pulling latest docker.io/ceph/ceph:v15 container...
INFO:cephadm:Extracting ceph user uid/gid from container image...
INFO:cephadm:Creating initial keys...
INFO:cephadm:Creating initial monmap...
INFO:cephadm:Creating mon...
INFO:cephadm:Waiting for mon to start...
INFO:cephadm:Waiting for mon...
INFO:cephadm:Assimilating anything we can from ceph.conf...
INFO:cephadm:Generating new minimal ceph.conf...
INFO:cephadm:Restarting the monitor...
INFO:cephadm:Setting mon public_network...
INFO:cephadm:Creating mgr...
INFO:cephadm:Wrote keyring to /etc/ceph/ceph.client.admin.keyring
INFO:cephadm:Wrote config to /etc/ceph/ceph.conf
INFO:cephadm:Waiting for mgr to start...
INFO:cephadm:Waiting for mgr...
INFO:cephadm:mgr not available, waiting (1/10)...
INFO:cephadm:mgr not available, waiting (2/10)...
INFO:cephadm:mgr not available, waiting (3/10)...
INFO:cephadm:Enabling cephadm module...
INFO:cephadm:Waiting for the mgr to restart...
INFO:cephadm:Waiting for Mgr epoch 5...
INFO:cephadm:Setting orchestrator backend to cephadm...
INFO:cephadm:Generating ssh key...
INFO:cephadm:Wrote public SSH key to to /etc/ceph/ceph.pub
INFO:cephadm:Adding key to root@localhost's authorized_keys...
INFO:cephadm:Adding host ceph01...
INFO:cephadm:Deploying mon service with default placement...
INFO:cephadm:Deploying mgr service with default placement...
INFO:cephadm:Deploying crash service with default placement...
INFO:cephadm:Enabling mgr prometheus module...
INFO:cephadm:Deploying prometheus service with default placement...
INFO:cephadm:Deploying grafana service with default placement...
INFO:cephadm:Deploying node-exporter service with default placement...
INFO:cephadm:Deploying alertmanager service with default placement...
INFO:cephadm:Enabling the dashboard module...
INFO:cephadm:Waiting for the mgr to restart...
INFO:cephadm:Waiting for Mgr epoch 13...
INFO:cephadm:Generating a dashboard self-signed certificate...
INFO:cephadm:Creating initial admin user...
INFO:cephadm:Fetching dashboard port number...
INFO:cephadm:Ceph Dashboard is now available at:

URL: https://localhost:8443/
User: admin
Password: mdbewc14gq

INFO:cephadm:You can access the Ceph CLI with:

sudo /usr/sbin/cephadm shell --fsid 7bffaaf6-9688-11ea-ac24-080027b4217f -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring

INFO:cephadm:Please consider enabling telemetry to help improve Ceph:

ceph telemetry on

For more information see:

https://docs.ceph.com/docs/master/mgr/telemetry/

INFO:cephadm:Bootstrap complete.

4、此时可以通过最后给出的信息,登录管理页面了

5、修改配置文件

sudo vi /etc/ceph/ceph.conf

[global]
fsid = a4547d9d-f1a1-4753-b5cc-df0e043ebc65
mon_initial_members = ceph01
#原本生成的mon_host好像有些问题
#mon_host = [v2:ceph01:3300/0,v1:ceph:6789/0]
mon_host = 172.16.172.101
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
public_network = 172.16.172.0/24

6、查看ceph状态

sudo ceph status
cluster:
id:     7bffaaf6-9688-11ea-ac24-080027b4217f
health: HEALTH_WARN
Reduced data availability: 1 pg inactive
OSD count 0 < osd_pool_default_size 3

services:
mon: 1 daemons, quorum ceph01 (age 35m)
mgr: ceph01.lreqdw(active, since 33m)
osd: 0 osds: 0 up, 0 in

data:
pools:   1 pools, 1 pgs
objects: 0 objects, 0 B
usage:   0 B used, 0 B / 0 B avail
pgs:     100.000% pgs unknown
1 unknown

7、三个子节点做好准备

#在ceph01
#将ceph.pub拷贝到其他三个节点
scp /etc/ceph/ceph.pub  neohope@ceph02:~/authorized_keys

#在ceph02
#启用root
sudo passwd -u root
#设置好root的authorized_keys
mv authorized_keys  /root/.ssh/
cd /root/.ssh/
chown root:root authorized_keys
chmod 0600 authorized_keys
#允许root ssh登录
sudo sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
sudo service ssh restart

#在ceph01
#获取私钥
sudo ceph config-key get mgr/cephadm/ssh_identity_key > ceph.pem
chmod 0600 ceph.pem
#测试root登录
ssh  -i ceph.pem root@ceph02

8、三个子节点加入网络

sudo ceph orch host add ceph02
Added host 'ceph02'

sudo ceph orch host add ceph03
Added host 'ceph03'

sudo ceph orch host add ceph04
Added host 'ceph04'

sudo ceph orch host ls
HOST    ADDR    LABELS  STATUS
ceph01  ceph01
ceph02  ceph02
ceph03  ceph03
ceph04  ceph04

9、设置mon

ceph orch apply mon 4
ceph orch apply mon ceph01,ceph02,ceph03,ceph04
sudo ceph status

ISTIO环境搭建02

换了3个云厂商,才把最后的例子跑完。。。

1、下载示例源码

git clone https://github.com/istio/istio.git
Cloning into 'istio'...

2、生成镜像

cd istio/samples/helloworld/src
./build_service.sh
Sending build context to Docker daemon  7.168kB
Step 1/8 : FROM python:2-onbuild
2-onbuild: Pulling from library/python
......

sudo docker images
REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
istio/examples-helloworld-v2         latest              2c7736ccfb8b        45 seconds ago      713MB
istio/examples-helloworld-v1         latest              20be3b24eab7        46 seconds ago      713MB

3、镜像发到其他节点

# 备份镜像
sudo docker save -o hello1.tar 20be3b24eab7
sudo docker save -o hello2.tar 2c7736ccfb8b

# 镜像发送到其他3个节点,并导入
# 对于每个节点做下面的操作
scp -i ~/hwk8s.pem hello1.tar root@192.168.1.229:~/
scp -i ~/hwk8s.pem hello2.tar root@192.168.1.229:~/

ssh -i 192.168.1.229

sudo docker load -i hello1.tar
sudo docker tag 20be3b24eab7 istio/examples-helloworld-v1:latest

sudo docker load -i hello2.tar
sudo docker tag 2c7736ccfb8b istio/examples-helloworld-v2:latest

exit

4、部署helloworld

kubectl apply -f helloworld.yaml
service/helloworld created
deployment.apps/helloworld-v1 created
deployment.apps/helloworld-v2 created

kubectl apply -f helloworld-gateway.yaml
gateway.networking.istio.io/helloworld-gateway created
virtualservice.networking.istio.io/helloworld created

kubectl get pods
kubectl get deployments

5、测试并生成流量

# 设置环境变量
# 这里IP要选用内网IP
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')
export GATEWAY_URL=192.168.1.124:$INGRESS_PORT

# 测试一下,会发现两次访问会用不同版本的服务
curl http://$GATEWAY_URL/hello
Hello version: v1, instance: helloworld-v1-5b75657f75-9dss5
curl http://$GATEWAY_URL/hello
Hello version: v2, instance: helloworld-v2-7855866d4f-rd2tr

# 也可以在外网,通过浏览器浏览
# 这里IP要选用外网IP
# 同样的,刷新浏览器会在不同版本服务之间切换
http://159.138.135.216:INGRESS_PORT/hello

# 生成访问流量
./loadgen.sh

6、使用kiali查看相关信息

#kiali 20001
istioctl dashboard kiali

#按上一节方法修改nginx配置内部端口
#重新加载配置
nginx -s reload

# 浏览器访问
http://159.138.135.216:8000

7、其他dashboard信息也可以用相同方法访问

#grafana 3000
istioctl dashboard grafana
#jaeger  16686
istioctl dashboard jaeger
#kiali 20001
istioctl dashboard kiali
#prometheus 9090
istioctl dashboard prometheus
#podid 9876
istioctl dashboard controlz podid
#podid 15000
istioctl dashboard envoy podid
#zipkin
istioctl dashboard zipkin

#按上一节方法修改nginx配置内部端口
#重新加载配置
nginx -s reload

# 浏览器访问
http://159.138.135.216:8000

ISTIO环境搭建01

1、首先,请根据前面k8s的教程,搭建一套可以运行的k8s环境
搭建Kubernetes环境01

搭建Kubernetes环境02

k8s-0001 159.138.135.216 192.168.1.124
k8s-0002 159.138.139.37 192.168.1.229
k8s-0003 159.138.31.39 192.168.1.187
k8s-0004 119.8.113.135 192.168.1.83

2、下载并部署istio

#下载并部署istio
curl -L https://istio.io/downloadIstio | sh -
cd istio-1.5.2
export PATH=$PWD/bin:$PATH
istioctl manifest apply --set profile=demo
Detected that your cluster does not support third party JWT authentication. Falling back to less secure first party JWT. See https://istio.io/docs/ops/best-practices/security/#configure-third-party-service-account-tokens for details.
- Applying manifest for component Base...
✔ Finished applying manifest for component Base.
- Applying manifest for component Pilot...
✔ Finished applying manifest for component Pilot.
Waiting for resources to become ready...
Waiting for resources to become ready...
Waiting for resources to become ready...
Waiting for resources to become ready...
Waiting for resources to become ready...
- Applying manifest for component EgressGateways...
- Applying manifest for component IngressGateways...
- Applying manifest for component AddonComponents...
✔ Finished applying manifest for component EgressGateways.
✔ Finished applying manifest for component AddonComponents.
✔ Finished applying manifest for component IngressGateways.
✔ Installation complete

#告知istio,对default空间下的pod自动注入Envoy sidecar
kubectl label namespace default istio-injection=enabled
namespace/default labeled

3、部署demo

#部署
kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml

#查看pods情况
kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
details-v1-6fc55d65c9-kxxpm       2/2     Running   0          106s
productpage-v1-7f44c4d57c-h6h7p   2/2     Running   0          105s
ratings-v1-6f855c5fff-2rjz9       2/2     Running   0          105s
reviews-v1-54b8794ddf-tq5vm       2/2     Running   0          106s
reviews-v2-c4d6568f9-q8mvs        2/2     Running   0          106s
reviews-v3-7f66977689-ccp9c       2/2     Running   0          106s

#查看services情况
kubectl get services
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
details       ClusterIP   10.104.68.235   <none>        9080/TCP   89s
kubernetes    ClusterIP   10.96.0.1       <none>        443/TCP    31m
productpage   ClusterIP   10.106.255.85   <none>        9080/TCP   89s
ratings       ClusterIP   10.103.19.155   <none>        9080/TCP   89s
reviews       ClusterIP   10.110.79.44    <none>        9080/TCP   89s</none></none></none></none></none>

# 开启外部访问
kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml
gateway.networking.istio.io/bookinfo-gateway created
virtualservice.networking.istio.io/bookinfo created

#查看gateway情况
kubectl get gateway
NAME               AGE
bookinfo-gateway   7s

4、设置ingress

# 查看是否配置了外部IP
kubectl get svc istio-ingressgateway -n istio-system
NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                                                                                      AGE
istio-ingressgateway   LoadBalancer   10.105.220.60   <pending>     15020:32235/TCP,80:30266/TCP,443:30265/TCP,15029:30393/TCP,15030:30302/TCP,15031:30789/TCP,15032:31411/TCP,31400:30790/TCP,15443:31341/TCP   5m30s</pending>

#使用node的地址作为host,和LB只需要配置一种
export INGRESS_HOST=47.57.158.253

#使用LB的地址作为host,和node只需要配置一种
export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

#配置http端口
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')

#配置https端口
export SECURE_INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="https")].nodePort}')

#设置并查看外部访问地址
export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT
echo http://$GATEWAY_URL/productpage

#此时就可以通过节点的ip地址来访问部署的实例了
#浏览器打开上面输出的地址
#http://47.57.158.253:30266/productpage

5、开启管理页面

#开始kaili
istioctl dashboard kiali

#安装nginx
#并设置反向代理
vi /etc/nginx/nginx.conf
http {

  upstream backend {
    # 代理的本地端口
    server 127.0.0.1:20001;
  }

  server {
    # 访问的外部端口
    listen 8000;
    location / {
      proxy_pass http://backend;
    }
  }

}

# 通过反向代理的8000端口就可以访问kiali的管理界面了
# #http://47.57.158.253:8000

PS:
必须开放的TCP端口有:

8000 nginx代理端口
8001 k8s默认代理端口
30266 bookinfo demo端口,会变更

Ubuntu18升级到20

1、升级准备
做好资料备份

2、开始升级

# 更新Ubuntu18
sudo apt get update
sudo apt get upgrade
sudo apt autoremove

# 查看有哪些版本可以升级
sudo do-release-upgrade -c
sudo do-release-upgrade -d -c

# 升级到Ubuntu20
sudo do-release-upgrade -d

# 系统会自动升级,升级后,桌面操作快乐很多
# 现阶段有个问题,就是升级后,桌面快捷方式都不能使用了,不知道怎么回事

3、替换国内源

sudo vi /etc/apt/sources.list
deb http://mirrors.163.com/ubuntu/ focal main restricted universe multiverse
deb http://mirrors.163.com/ubuntu/ focal-security main restricted universe multiverse
deb http://mirrors.163.com/ubuntu/ focal-updates main restricted universe multiverse
deb http://mirrors.163.com/ubuntu/ focal-proposed main restricted universe multiverse
deb http://mirrors.163.com/ubuntu/ focal-backports main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ focal main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ focal-security main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ focal-updates main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ focal-proposed main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ focal-backports main restricted universe multiverse

4、禁用自动更新

# 1都改为0
sudo vi /etc/apt/apt.conf.d/10periodic
APT::Periodic::Update-Package-Lists "0";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
APT::Periodic::Unattended-Upgrade "0";

V8引擎常用命令

1、测试文件test.js

function log(msg){
    console.log(msg)
}

hi = "hello"
log(hi)

2、输出AST语法树

d8 --print-ast test.js
[generating bytecode for function: ]
--- AST ---
FUNC at 0
. KIND 0
. LITERAL ID 0
. SUSPEND COUNT 0
. NAME ""
. INFERRED NAME ""
. DECLS
. . FUNCTION "log" = function log
. EXPRESSION STATEMENT at 47
. . ASSIGN at 50
. . . VAR PROXY unallocated (0000022CC5A21D98) (mode = DYNAMIC_GLOBAL, assigned = true) "hi"
. . . LITERAL "hello"
. EXPRESSION STATEMENT at 61
. . ASSIGN at -1
. . . VAR PROXY local[0] (0000022CC5A21CC8) (mode = TEMPORARY, assigned = true) ".result"
. . . CALL
. . . . VAR PROXY unallocated (0000022CC5A21B10) (mode = VAR, assigned = true) "log"
. . . . VAR PROXY unallocated (0000022CC5A21D98) (mode = DYNAMIC_GLOBAL, assigned = true) "hi"
. RETURN at -1
. . VAR PROXY local[0] (0000022CC5A21CC8) (mode = TEMPORARY, assigned = true) ".result"

[generating bytecode for function: log]
--- AST ---
FUNC at 12
. KIND 0
. LITERAL ID 1
. SUSPEND COUNT 0
. NAME "log"
. PARAMS
. . VAR (0000022CC5A21B68) (mode = VAR, assigned = false) "msg"
. DECLS
. . VARIABLE (0000022CC5A21B68) (mode = VAR, assigned = false) "msg"
. EXPRESSION STATEMENT at 24
. . CALL
. . . PROPERTY at 32
. . . . VAR PROXY unallocated (0000022CC5A21D18) (mode = DYNAMIC_GLOBAL, assigned = false) "console"
. . . . NAME log
. . . VAR PROXY parameter[0] (0000022CC5A21B68) (mode = VAR, assigned = false) "msg"

hello

3、输出作用域

d8 --print-scopes test.js
Inner function scope:
function log () { // (000001E067B50EA8) (12, 43)
  // 2 heap slots
  // local vars:
  VAR msg;  // (000001E067B54E18) never assigned
}
Global scope:
global { // (000001E067B50C88) (0, 68)
  // will be compiled
  // 1 stack slots
  // temporary vars:
  TEMPORARY .result;  // (000001E067B51288) local[0]
  // local vars:
  VAR log;  // (000001E067B510D0)
  // dynamic vars:
  DYNAMIC_GLOBAL hi;  // (000001E067B51358)

  function log () { // (000001E067B50EA8) (12, 43)
    // lazily parsed
    // 2 heap slots
  }
}
Global scope:
function log (msg) { // (000001E067B50EA8) (12, 43)
  // will be compiled
  // local vars:
  VAR msg;  // (000001E067B51128) parameter[0], never assigned
}
hello

4、输出字节码

d8 --print-bytecode test.js
[generated bytecode for function:  (0x03970824fc39 <SharedFunctionInfo>)]
Parameter count 1
Register count 3
Frame size 24
         000003970824FCEA @    0 : 12 00             LdaConstant [0]
         000003970824FCEC @    2 : 26 fa             Star r1
         000003970824FCEE @    4 : 27 fe f9          Mov <closure>, r2
         000003970824FCF1 @    7 : 61 37 01 fa 02    CallRuntime [DeclareGlobals], r1-r2
         000003970824FCF6 @   12 : 12 01             LdaConstant [1]
         000003970824FCF8 @   14 : 15 02 00          StaGlobal [2], [0]
         000003970824FCFB @   17 : 13 03 02          LdaGlobal [3], [2]
         000003970824FCFE @   20 : 26 fa             Star r1
         000003970824FD00 @   22 : 13 02 04          LdaGlobal [2], [4]
         000003970824FD03 @   25 : 26 f9             Star r2
         000003970824FD05 @   27 : 5d fa f9 06       CallUndefinedReceiver1 r1, r2, [6]
         000003970824FD09 @   31 : 26 fb             Star r0
         000003970824FD0B @   33 : aa                Return
Constant pool (size = 4)
000003970824FCB1: [FixedArray] in OldSpace
 - map: 0x0397080404b1 <Map>
 - length: 4
           0: 0x03970824fc61 <FixedArray[2]>
           1: 0x03970824fc01 <String[#5]: hello>
           2: 0x03970824fbf1 <String[#2]: hi>
           3: 0x0397081c692d <String[#3]: log>
Handler Table (size = 0)
Source Position Table (size = 0)
[generated bytecode for function: log (0x03970824fc71 <SharedFunctionInfo log>)]
Parameter count 2
Register count 2
Frame size 16
         000003970824FE4E @    0 : 13 00 00          LdaGlobal [0], [0]
         000003970824FE51 @    3 : 26 fa             Star r1
         000003970824FE53 @    5 : 28 fa 01 02       LdaNamedProperty r1, [1], [2]
         000003970824FE57 @    9 : 26 fb             Star r0
         000003970824FE59 @   11 : 59 fb fa 02 04    CallProperty1 r0, r1, a0, [4]
         000003970824FE5E @   16 : 0d                LdaUndefined
         000003970824FE5F @   17 : aa                Return
Constant pool (size = 2)
000003970824FE1D: [FixedArray] in OldSpace
 - map: 0x0397080404b1 <Map>
 - length: 2
           0: 0x0397081c68b9 <String[#7]: console>
           1: 0x0397081c692d <String[#3]: log>
Handler Table (size = 0)
Source Position Table (size = 0)
hello

5、测试文件test1.js

let a = {x:1}

function bar(obj) { 
  return obj.x 
}

function foo (count) { 
  let ret = 0
  for(let i = 1; i < count; i++) {
    ret += bar(a)
  }
  return ret
}

//foo(7049)
//foo(100000)

6、输出优化信息

//foo(7049)
d8 --trace-opt-verbose test1.js
[not yet optimizing foo, not enough ticks: 0/2 and ICs changed]

//foo(10000)
[not yet optimizing foo, not enough ticks: 0/2 and ICs changed]
[marking 0x01410824fe35 <JSFunction foo (sfi = 000001410824FCD9)> for optimized recompilation, reason: small function]
[compiling method 0x01410824fe35 <JSFunction foo (sfi = 000001410824FCD9)> using TurboFan OSR]
[optimizing 0x01410824fe35 <JSFunction foo (sfi = 000001410824FCD9)> - took 141.370, 62.753, 1.551 ms]

7、输出反优化信息

//没有输出。。。
d8 --trace-deopt test1.js

8、test2.js

function strToArray(str) {
  let i = 0
  const len = str.length
  let arr = new Uint16Array(str.length)
  for (; i < len; ++i) {
    arr[i] = str.charCodeAt(i)
  }
  return arr;
}

function foo() {
  let i = 0
  let str = 'test V8 GC'
  while (i++ < 1e5) {
    strToArray(str);
  }
}

foo()

9、输出反优化信息

d8 --trace-gc test2.js
[4600:0000019D00000000]      490 ms: Scavenge 1.2 (2.4) -> 0.3 (3.4) MB, 14.5 / 0.0 ms  (average mu = 1.000, current mu = 1.000) allocation failure
[4600:0000019D00000000]      500 ms: Scavenge 1.2 (3.4) -> 0.3 (3.6) MB, 2.0 / 0.0 ms  (average mu = 1.000, current mu = 1.000) allocation failure
[4600:0000019D00000000]      501 ms: Scavenge 1.3 (3.6) -> 0.3 (3.6) MB, 0.1 / 0.0 ms  (average mu = 1.000, current mu = 1.000) allocation failure
[4600:0000019D00000000]      503 ms: Scavenge 1.3 (3.6) -> 0.3 (3.6) MB, 0.1 / 0.0 ms  (average mu = 1.000, current mu = 1.000) allocation failure
[4600:0000019D00000000]      505 ms: Scavenge 1.3 (3.6) -> 0.3 (3.6) MB, 0.1 / 0.0 ms  (average mu = 1.000, current mu = 1.000) allocation failure
[4600:0000019D00000000]      506 ms: Scavenge 1.3 (3.6) -> 0.3 (3.6) MB, 0.1 / 0.0 ms  (average mu = 1.000, current mu = 1.000) allocation failure
[4600:0000019D00000000]      508 ms: Scavenge 1.3 (3.6) -> 0.3 (3.6) MB, 0.1 / 0.0 ms  (average mu = 1.000, current mu = 1.000) allocation failure
[4600:0000019D00000000]      509 ms: Scavenge 1.3 (3.6) -> 0.3 (3.6) MB, 0.1 / 0.0 ms  (average mu = 1.000, current mu = 1.000) allocation failure
[4600:0000019D00000000]      511 ms: Scavenge 1.3 (3.6) -> 0.3 (3.6) MB, 0.1 / 0.0 ms  (average mu = 1.000, current mu = 1.000) allocation failure
[4600:0000019D00000000]      513 ms: Scavenge 1.3 (3.6) -> 0.3 (3.6) MB, 0.1 / 0.0 ms  (average mu = 1.000, current mu = 1.000) allocation failure
[4600:0000019D00000000]      515 ms: Scavenge 1.3 (3.6) -> 0.3 (3.6) MB, 0.1 / 0.0 ms  (average mu = 1.000, current mu = 1.000) allocation failure
[4600:0000019D00000000]      516 ms: Scavenge 1.3 (3.6) -> 0.3 (3.6) MB, 0.1 / 0.0 ms  (average mu = 1.000, current mu = 1.000) allocation failure
[4600:0000019D00000000]      518 ms: Scavenge 1.3 (3.6) -> 0.3 (3.6) MB, 0.1 / 0.0 ms  (average mu = 1.000, current mu = 1.000) allocation failure
[4600:0000019D00000000]      520 ms: Scavenge 1.3 (3.6) -> 0.3 (3.6) MB, 0.1 / 0.0 ms  (average mu = 1.000, current mu = 1.000) allocation failure

10、test3.js

function Foo(property_num,element_num) {
  //添加可索引属性
  for (let i = 0; i < element_num; i++) {
      this[i] = `element${i}`
  }
  //添加常规属性
  for (let i = 0; i < property_num; i++) {
      let ppt = `property${i}`
      this[ppt] = ppt
  }
}
var bar = new Foo(10,10)
console.log(%HasFastProperties(bar));
delete bar.property2
console.log(%HasFastProperties(bar));

11、使用内部方法

//%HasFastProperties测试是否有快属性
d8 --allow-natives-syntax test3.js
true
false