TensorFlow入门02:Tensor

Tensorflow中所有的数据都称为Tensor,可以是一个变量、数组或者多维数组。Tensor 有几个重要的属性:
Rank:纬数,比如scalar rank=0, vector rank=1, matrix rank=2
Shape:形状,比如vector shape=[D0], matrix shape=[D0, D1]
类型:数据类型,比如tf.float32, tc.uint8等

Rank与Shape关系如下表所示

Rank Shape Dimension number Example
0 [] 0-D A 0-D tensor. A scalar.
1 [D0] 1-D A 1-D tensor with shape [5].
2 [D0, D1] 2-D A 2-D tensor with shape [3, 4].
3 [D0, D1, D2] 3-D A 3-D tensor with shape [1, 4, 3].
n [D0, D1, … Dn-1] n-D A tensor with shape [D0, D1, … Dn-1].

TensorFlow入门01:环境搭建

1、CPU版本安装

1.1 安装tensorflow

pip3 install --upgrade tensorflow

1.2 Python验证,看到版本信息就可以了

python3
>>> import tensorflow as tf
>>> print('Tensorflow version ', tf.__version__)

Tensorflow version  1.12.0

2、GPU版本安装(需要NVIDIA显卡)

2.1 检查驱动信息

nvidia-smi

Fri Nov 16 21:22:13 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.77                 Driver Version: 390.77                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 106...  Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   52C    P2    27W /  N/A |   5938MiB /  6078MiB |     22%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      7953      G   /usr/lib/xorg/Xorg                           126MiB |
|    0      8215      G   /usr/bin/gnome-shell                         109MiB |
|    0     13578    C+G   python3                                     5689MiB |
+-----------------------------------------------------------------------------+

2.2 安装CUDA

# 查看网站 https://developer.nvidia.com/cuda-90-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1704&target_type=runfilelocal
# 选择下载这个版本 Linux x86_64 Ubuntu 17.04 runfile
# 安装,但注意不要更新驱动
sudo chmod +x cuda_9.0.176_384.81_linux.run
./cuda_9.0.176_384.81_linux.run --override

2.3 安装CUDNN

# 查看网站 https://developer.nvidia.com/rdp/cudnn-download
# 选择下载这个版本 9.0 cuDNN Library for Linux
# 解压
tar -zxvf cudnn-9.0-linux-x64-v7.tgz
# 手工安装
sudo cp -P cuda/lib64/libcudnn* /usr/local/cuda-9.0/lib64/
sudo cp  cuda/include/cudnn.h /usr/local/cuda-9.0/include/
# 调整权限
sudo chmod a+r /usr/local/cuda-9.0/include/cudnn.h /usr/local/cuda-9.0/lib64/libcudnn*

2.3 安装libcupti-dev

sudo apt-get install libcupti-dev

2.4 修改.bashrc

# 增加下面两行
export PATH=/usr/local/cuda-9.0/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

2.5 安装tensorflow-gpu

pip3 install --upgrade tensorflow-gpu

2.6 Python验证,看到GPU就可以啦

Python3
>>> from tensorflow.python.client import device_lib
>>> device_lib.list_local_devices()

[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
...
incarnation: 2559160109308400478
physical_device_desc: "device: 0, name: GeForce GTX 1060 with Max-Q Design, pci bus id: 0000:01:00.0, compute capability: 6.1"
]

3、Docker方式安装
3.1 CPU版

# 运行tensorflow
docker run -it -p 8888:8888 gcr.io/tensorflow/tensorflow

3.2 GPU版

# 安装nvidia-docker
wget https://github.com/NVIDIA/nvidia-docker/releases/download/v1.0.1/nvidia-docker_1.0.1-1_amd64.deb
sudo dpkg -i nvidia-docker*.deb

# 测试nvidia-docker,执行nvidia-smi命令
nvidia-docker run --rm nvidia/cuda nvidia-smi

# 运行tensorflow
nvidia-docker run -it -p 8888:8888 gcr.io/tensorflow/tensorflow:latest-gpu

4、编译CUDA Demo(非必须)

# 咱们选用的版本只支持到gcc6
apt-get install gcc-6 g++-6
ln -s /bin/gcc /bin/gcc-6

# 安装libmpich-dev
sudo apt-get install libmpich-dev


# 切换路径
cd PATH_TO_DEMO

# 编译
make

Ubuntu18.0.4LTS蓝牙鼠标无法重连

Ubuntu18.0.4LTS安装后,蓝牙鼠标经常无法重连,休眠、重启后,都需要重新配对连接。现在咱们来解决这个问题。

1、首先删掉配对的鼠标

2、打开Terminal

bluetoothctl

# 查看蓝牙控制器,一般只有一个
>>> list

# 选择蓝牙控制器
>>> select 01:23:45:67:89:AB

# 查看控制器情况
>>> show

# 启动
>>> power on

# 扫描,记录蓝牙鼠标地址
>>> scan on

# 关闭扫描
>>> scan off

# 开启agent
>>> agent on

# 配对
>>> pair 34:88:5D:87:C0:A6

# 连接
>>> connect 34:88:5D:87:C0:A6

# 信任设备
>>> trust 34:88:5D:87:C0:A6

# 退出

3、 再试一下,是不是好了

windows下无法启动pip

尤其是在有多版本Python共存的情况下,修改windows修改环境变量后,经常会导致pip无法启动的情况。
此时,不仅是pip,Python/Scripts目录下的所有脚本都无法启动,并会有如下错误:

Fatal error in launcher: Unable to create process using '"'

其根本原因,其实十分简单,pip无法找到python.exe可执行程序,你可以看pip的源码来确认这一点。
有几种方法可以解决这个问题:

1、环境变量法,更适合单Ptyhon环境
将python.exe路径,增加到PATH环境变量中即可解决问题

2、脚本启动法,适合多个Ptyhon环境

set PATH=PATH_TO_PYTHON\;PATH_TO_PYTHON\Scripts;%PATH%
python -m pip install XXX

3、用1或2,更新pip,可以解决问题(对单Python环境更适用)

python -m pip install --upgrade pip

4、修改pip二进制文件
用十六进制编辑工具打开pip.exe
修改python.exe路径
保存

5、用PE编辑器修改pip二进制文件
同方法4

6、解压
用解压工具解压pip,
得到__main__.py
重命名为pip.py
运行

python pip.py install XXX

知识图谱03:JENA

1、下载apache-jena-fuseki和apache-jena
https://jena.apache.org/download/index.cgi

2、将上一篇教程的nt文件转换为tdb格式

cd apache-jena-3.9.0\bat
tdbloader.bat --loc="PATH_TO_TDB\tdb" "PATH_TO_NT\movies_mapping.nt"

3、切换到apache-jena-fuseki-3.9.0目录,启动一次服务,然后退出

4、将教程1里面的Movies.owl,拷贝到apache-jena-fuseki-3.9.0\run\databases路径下面,并重命名为Movies.ttl

5、创建配置文件apache-jena-fuseki-3.9.0\run\configuration\fuseki_conf.ttl

@prefix fuseki: <http://jena.apache.org/fuseki#> . 
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> . 
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . 
@prefix tdb: <http://jena.hpl.hp.com/2008/tdb#> . 
@prefix ja: <http://jena.hpl.hp.com/2005/11/Assembler#> . 
@prefix : <http://base/#> . 

<#service> rdf:type fuseki:Service ; 
    fuseki:name "movies" ;
    fuseki:serviceQuery "sparql" ;
    fuseki:dataset <#dataset> ; 
    fuseki:serviceReadGraphStore      "get" ;
    fuseki:serviceReadWriteGraphStore "data" ;
    fuseki:serviceUpdate              "update" ;
    fuseki:serviceUpload              "upload"
    . 

<#dataset> rdf:type ja:RDFDataset ;
	ja:defaultGraph <#modelInf> ;
	.

<#modelInf> 
    rdf:type ja:InfModel ;
    #ja:reasoner [ja:reasonerURL <http://jena.hpl.hp.com/2003/OWLFBRuleReasoner>]  
    ja:reasoner [ 
        ja:reasonerURL <http://jena.hpl.hp.com/2003/GenericRuleReasoner> ; 
        ja:rulesFrom <file:///D:/ProjectsMy/KG/apache-jena-fuseki-3.9.0/run/databases/Rules.ttl> ] ; 
    ja:baseModel <#baseModel> ; 
    . 

<#baseModel> rdf:type tdb:GraphTDB ; 
    tdb:location "D:/ProjectsMy/KG/workspace/data/tdb" ; 
    tdb:unionDefaultGraph true ; 
    .

6、创建规则文件apache-jena-fuseki-3.9.0\run\databases\Movies.ttl
这个规则规定了,演过喜剧的演员,叫做喜剧演员(Comedian)

@prefix xsd: <XML Schema> .
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix : <http://www.neohope.com/hansen/ontologies/2018/movies#> .

[ruleComedian: (?aPerson :hasActedIn ?aMovie) (?aMovie :hasGenre ?aGenre) (?aGenre :genreName '喜剧') -> (?aPerson rdf:type :Comedian)]
[ruleInverse: (?aPerson :hasActedIn ?aMove) -> (?aMovie :hasActor ?aPerson)]

7、启动apache-jena-fuseki-3.9.0

8、访问http://localhost:3030/

9、进行查询,上一篇的例子也都可以用
http://localhost:3030/dataset.html?tab=query&ds=/movies

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
prefix owl: <http://www.w3.org/2002/07/owl#>
prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>
prefix : <http://www.neohope.com/hansen/ontologies/2018/movies#>

SELECT ?name WHERE {
?aComedian rdf:type :Comedian.
?aComedian :personName ?name.
}
LIMIT 10

10、通过python访问
https://github.com/neohope/kg-demo-for-movie/tree/master/src/query-jena.py

参考链接:
https://zhuanlan.zhihu.com/knowledgegraph
https://github.com/SimmerChan/KG-demo-for-movie

PS:
参考教程中,原作者通过结巴分词+正则匹配+Jena,实现了一个简单的问答系统,感兴趣的话,大家可以看下。

知识图谱02:RDF

1、安装MySQL5,并新建movies库

2、导入数据
https://github.com/neohope/kg-demo-for-movie/tree/master/data/movies.sql

3、下载d2rq,并配置好JDK环境变量
http://d2rq.org/

4、利用d2rq生成mapping

generate-mapping -u movie -p password -o movies_mapping.ttl jdbc:mysql:///movies

5、手工编辑ttl,任务如下
设置正确的域名
修正类名与属性名
删除一些不需要的字段
修改前后的数据可以在这里找到
https://github.com/neohope/kg-demo-for-movie/tree/master/data/movies_mapping.ttl
https://github.com/neohope/kg-demo-for-movie/tree/master/data/movies_mapping_ok.ttl

6、输出RDF文件,用于后续的教程

dump-rdf.bat -o movies_mapping.nt movies_mapping_ok.ttl

7、启动d2r服务

d2r-server.bat movies_mapping_ok.ttl

8、访问及浏览数据
http://localhost:2020/

9、查询
http://localhost:2020/snorql/

#周星驰演过的电影
SELECT ?title WHERE {
  ?aPerson rdf:type :Person.
  ?aPerson :personName '周星驰'.
  ?aPerson :hasActedIn ?aMovie.
  ?aMovie :movieTitle ?title
}
LIMIT 10


#英雄的演员
SELECT ?actor WHERE {
  ?aMovie rdf:type :Movie.
  ?aMovie :movieTitle '英雄'.
  ?aPerson :hasActedIn ?aMovie.
  ?aPerson :personName ?actor
}
LIMIT 10


#巩俐参演的,评分高于7的电影
SELECT ?title WHERE {
  ?aPerson rdf:type :Person.
  ?aPerson  :personName '巩俐'.
  ?aPerson  :hasActedIn ?aMovie.
  ?aMovie :movieTitle ?title.
  ?aMovie :movieRating ?rating.
  FILTER (?rating>=7)
}
LIMIT 10

10、通过python访问
https://github.com/neohope/kg-demo-for-movie/tree/master/src/query-d2rq.py

参考链接:
https://zhuanlan.zhihu.com/knowledgegraph
https://github.com/SimmerChan/KG-demo-for-movie

知识图谱01:本体建模

1、下载Protege工具

https://protege.stanford.edu/

2、安装JDK,并在配置好JDK环境变量

3、打开Protege

4、在Active Ontology页面,填写两个IRI,我分别填写了下面的数值

#Ontology IRI
http://www.neohope.com/hansen/ontologies/2018/movies
#Ontology Version IRI
http://www.neohope.com/hansen/ontologies/2018/movies/1.0.0

5、在Entities页面,切换到Classes,新建三个Class

Genre
Movie
Person

6、Entities页面,切换到Data properties,新建以下属性

genereId{Domain=Genre,Ranges=xsd:string}
genereName{Domain=Genre,Ranges=xsd:string}
movieId{Domain=Movie,Ranges=xsd:string}
movieIntroduction{Domain=Movie,Ranges=xsd:string}
movieRating{Domain=Movie,Ranges=xsd:string}
movieReleaseDate{Domain=Movie,Ranges=xsd:string}
movieTitile{Domain=Movie,Ranges=xsd:string}
personAppellation{Domain=Person,Ranges=xsd:string}
->personEnglishName{Domain=Person,Ranges=xsd:string}
->personName{Domain=Person,Ranges=xsd:string}
personBiography{Domain=Person,Ranges=xsd:string}
personbirthDay{Domain=Person,Ranges=xsd:string}
personBirthPlace{Domain=Person,Ranges=xsd:string}
personDeathDay{Domain=Person,Ranges=xsd:string}
personId{Domain=Person,Ranges=xsd:string}

7、Entities页面,切换到Object Properties,新建以下属性

hasActedIn{Domain=Person,Range=Movie,InverseOf=hasActor}
hasActor{Domain=Movie,Range=Person}
hasGenre{Domain=Person,Range=Genre}

8、保存为Movies.owl,这个文件可以在后面jena的例子中用到

9、建模后的结果,可以在这里获取:
https://github.com/neohope/kg-demo-for-movie/tree/master/protege

参考链接:
https://zhuanlan.zhihu.com/knowledgegraph
https://github.com/SimmerChan/KG-demo-for-movie

搭建WebRTC视频会议服务04

本节说明一下遗留的一些内容:

1、开启GPU加速

如果需要GPU加速,除了Intel指定的硬件外,还需要:
https://software.intel.com/en-us/intel-media-server-studio
有社区版,大家感兴趣的可以自己试一下。

2、开启VCA加速

如果需要支持VCA(Intel Visual Compute Accelerator)卡加速,请参考
https://software.intel.com/en-us/webrtc-conference-server-guide-mcu-installation#section2_3_10

3、支持集群

如果需要支持集群,请参考
https://software.intel.com/en-us/webrtc-conference-server-guide-mcu-installation#section2_3_8

咋说呢,总体上感觉这个集群方案有些过于复杂,需要用的时候再看吧。