Ubuntu18搭建CDH6环境03

1、确保cdt01可以ssh联通cdt02和cdt03

#这个userid与可以无密码使用sudo的userid相同
ssh -l userid cdh02
ssh -l userid cdh03

2、浏览器访问(以后都是界面了)
http://172.16.172.101:7180
用户名:admin
密码:admin

3、根据引导界面,新建Cluster
将172.16.172.101-172.16.172.103都安装好cloudera-manager-agent

4、根据引导界面,选用需要的软件进行安装
安装时,注意合理分配角色,也就是合理分配内存资源

5、依次安装
hdfs
zookeeper
hbase
yarn
hive
spark

6、安装完毕

PS:
1、如果出现找不到jdbc driver的情况

sudo apt-get install libmysql-java

Ubuntu18搭建CDH6环境02

1、cdt01安装

#添加cloudera仓库
wget https://archive.cloudera.com/cm6/6.3.0/ubuntu1804/apt/archive.key
sudo apt-key add archive.key
wget https://archive.cloudera.com/cm6/6.3.0/ubuntu1804/apt/cloudera-manager.list
sudo mv cloudera-manager.list /etc/apt/sources.list.d/

#更新软件清单
sudo apt-get update

#安装jdk8
sudo apt-get install openjdk-8-jdk

#安装cloudera
sudo apt-get install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

2、安装及配置mysql
2.1、安装mysql

sudo apt-get install mysql-server mysql-client libmysqlclient-dev libmysql-java

2.2、停止mysql

sudo service mysql stop

2.3、删除不需要的文件

sudo rm /var/lib/mysql/ib_logfile0
sudo rm /var/lib/mysql/ib_logfile1

2.4、修改配置文件

sudo vi /etc/mysql/mysql.conf.d/mysqld.cnf

#修改或添加以下信息
[mysqld]
transaction-isolation = READ-COMMITTED
max_allowed_packet = 32M
max_connections = 300
innodb_flush_method = O_DIRECT

2.5、启动mysql

sudo service mysql start

2.6、初始化mysql

sudo mysql_secure_installation

3、创建数据库并授权

sudo mysql -uroot -p
-- 创建数据库
-- Cloudera Manager Server
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Activity Monitor
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Reports Manager
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Hue
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Hive Metastore Server
CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Sentry Server
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Cloudera Navigator Audit Server
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Cloudera Navigator Metadata Server
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Oozie
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

#创建用户并授权
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm123456';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon123456';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman123456';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue123456';
GRANT ALL ON hive.* TO 'hive'@'%' IDENTIFIED BY 'hive123456';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry123456';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'nav123456';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'navms123456';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie123456';

4、初始化数据库

sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm scm123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql amon amon amon123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql rman rman rman123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql hue hue hue123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql hive hive hive123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql sentry sentry sentry123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql nav nav nav123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql navms navms navms123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql oozie oozie oozie123456

5、启动

#启动cloudera-scm-server
sudo systemctl start cloudera-scm-server

#查看启动日志,等待Jetty启动完成
sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

6、启动
浏览器访问
http://172.16.172.101:7180
用户名:admin
密码:admin

Ubuntu18搭建CDH6环境01

1、环境准备

VirtualBox 6
Ubuntu 18
Cloudera CDH 6.3

2、虚拟机安装Ubuntu18,配置为
1CPU
4G内存
300G硬盘
两块网卡,一块为HostOnly,一块为NAT

3、将虚拟机克隆为三份
如果是手工拷贝,记得修改硬盘UUID、虚拟机UUID、网卡硬件ID

4、设置IP地址、hostname及hosts文件

机器名 HostOnly IP
cdh01 172.16.172.101
cdh02 172.16.172.102
cdh03 172.16.172.103

5、允许无密码使用sudo,至少修改cdh02和cdh03

#edit /etc/sudoers
userid ALL=(ALL:ALL) NOPASSWD: ALL

TensorFlow入门02:Tensor

Tensorflow中所有的数据都称为Tensor,可以是一个变量、数组或者多维数组。Tensor 有几个重要的属性:
Rank:纬数,比如scalar rank=0, vector rank=1, matrix rank=2
Shape:形状,比如vector shape=[D0], matrix shape=[D0, D1]
类型:数据类型,比如tf.float32, tc.uint8等

Rank与Shape关系如下表所示

Rank Shape Dimension number Example
0 [] 0-D A 0-D tensor. A scalar.
1 [D0] 1-D A 1-D tensor with shape [5].
2 [D0, D1] 2-D A 2-D tensor with shape [3, 4].
3 [D0, D1, D2] 3-D A 3-D tensor with shape [1, 4, 3].
n [D0, D1, … Dn-1] n-D A tensor with shape [D0, D1, … Dn-1].

TensorFlow入门01:环境搭建

1、CPU版本安装

1.1 安装tensorflow

pip3 install --upgrade tensorflow

1.2 Python验证,看到版本信息就可以了

python3
>>> import tensorflow as tf
>>> print('Tensorflow version ', tf.__version__)

Tensorflow version  1.12.0

2、GPU版本安装(需要NVIDIA显卡)

2.1 检查驱动信息

nvidia-smi

Fri Nov 16 21:22:13 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.77                 Driver Version: 390.77                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 106...  Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   52C    P2    27W /  N/A |   5938MiB /  6078MiB |     22%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      7953      G   /usr/lib/xorg/Xorg                           126MiB |
|    0      8215      G   /usr/bin/gnome-shell                         109MiB |
|    0     13578    C+G   python3                                     5689MiB |
+-----------------------------------------------------------------------------+

2.2 安装CUDA

# 查看网站 https://developer.nvidia.com/cuda-90-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1704&target_type=runfilelocal
# 选择下载这个版本 Linux x86_64 Ubuntu 17.04 runfile
# 安装,但注意不要更新驱动
sudo chmod +x cuda_9.0.176_384.81_linux.run
./cuda_9.0.176_384.81_linux.run --override

2.3 安装CUDNN

# 查看网站 https://developer.nvidia.com/rdp/cudnn-download
# 选择下载这个版本 9.0 cuDNN Library for Linux
# 解压
tar -zxvf cudnn-9.0-linux-x64-v7.tgz
# 手工安装
sudo cp -P cuda/lib64/libcudnn* /usr/local/cuda-9.0/lib64/
sudo cp  cuda/include/cudnn.h /usr/local/cuda-9.0/include/
# 调整权限
sudo chmod a+r /usr/local/cuda-9.0/include/cudnn.h /usr/local/cuda-9.0/lib64/libcudnn*

2.3 安装libcupti-dev

sudo apt-get install libcupti-dev

2.4 修改.bashrc

# 增加下面两行
export PATH=/usr/local/cuda-9.0/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

2.5 安装tensorflow-gpu

pip3 install --upgrade tensorflow-gpu

2.6 Python验证,看到GPU就可以啦

Python3
>>> from tensorflow.python.client import device_lib
>>> device_lib.list_local_devices()

[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
...
incarnation: 2559160109308400478
physical_device_desc: "device: 0, name: GeForce GTX 1060 with Max-Q Design, pci bus id: 0000:01:00.0, compute capability: 6.1"
]

3、Docker方式安装
3.1 CPU版

# 运行tensorflow
docker run -it -p 8888:8888 gcr.io/tensorflow/tensorflow

3.2 GPU版

# 安装nvidia-docker
wget https://github.com/NVIDIA/nvidia-docker/releases/download/v1.0.1/nvidia-docker_1.0.1-1_amd64.deb
sudo dpkg -i nvidia-docker*.deb

# 测试nvidia-docker,执行nvidia-smi命令
nvidia-docker run --rm nvidia/cuda nvidia-smi

# 运行tensorflow
nvidia-docker run -it -p 8888:8888 gcr.io/tensorflow/tensorflow:latest-gpu

4、编译CUDA Demo(非必须)

# 咱们选用的版本只支持到gcc6
apt-get install gcc-6 g++-6
ln -s /bin/gcc /bin/gcc-6

# 安装libmpich-dev
sudo apt-get install libmpich-dev


# 切换路径
cd PATH_TO_DEMO

# 编译
make

Ubuntu18.0.4LTS蓝牙鼠标无法重连

Ubuntu18.0.4LTS安装后,蓝牙鼠标经常无法重连,休眠、重启后,都需要重新配对连接。现在咱们来解决这个问题。

1、首先删掉配对的鼠标

2、打开Terminal

bluetoothctl

# 查看蓝牙控制器,一般只有一个
>>> list

# 选择蓝牙控制器
>>> select 01:23:45:67:89:AB

# 查看控制器情况
>>> show

# 启动
>>> power on

# 扫描,记录蓝牙鼠标地址
>>> scan on

# 关闭扫描
>>> scan off

# 开启agent
>>> agent on

# 配对
>>> pair 34:88:5D:87:C0:A6

# 连接
>>> connect 34:88:5D:87:C0:A6

# 信任设备
>>> trust 34:88:5D:87:C0:A6

# 退出

3、 再试一下,是不是好了

windows下无法启动pip

尤其是在有多版本Python共存的情况下,修改windows修改环境变量后,经常会导致pip无法启动的情况。
此时,不仅是pip,Python/Scripts目录下的所有脚本都无法启动,并会有如下错误:

Fatal error in launcher: Unable to create process using '"'

其根本原因,其实十分简单,pip无法找到python.exe可执行程序,你可以看pip的源码来确认这一点。
有几种方法可以解决这个问题:

1、环境变量法,更适合单Ptyhon环境
将python.exe路径,增加到PATH环境变量中即可解决问题

2、脚本启动法,适合多个Ptyhon环境

set PATH=PATH_TO_PYTHON\;PATH_TO_PYTHON\Scripts;%PATH%
python -m pip install XXX

3、用1或2,更新pip,可以解决问题(对单Python环境更适用)

python -m pip install --upgrade pip

4、修改pip二进制文件
用十六进制编辑工具打开pip.exe
修改python.exe路径
保存

5、用PE编辑器修改pip二进制文件
同方法4

6、解压
用解压工具解压pip,
得到__main__.py
重命名为pip.py
运行

python pip.py install XXX