数据类型

用Python进行gRPC接口测试（三）

 https://mp.weixin.qq.com/s/eijIRBx-vKln0AftMqXkmQ

一、标量值类型

     标量值类型与我们在编程语言使用的基本数据类型概念类似，用来携带的数据也大体相同。在python中，这些标量值类型都能找到与之对应的python数据类型，处理起来简单便捷。

使用举例：

message Student {  string name = 1;  int32 age = 2;  // true: male, false:female  bool sex = 3;}

Python实现代码：

name="小王"age=15sex=True
#方式1student=Student(name=name,age=age,sex=sex)
#方式2student=Student()student.name=namestudent.age=agestudent.sex=sex

二、一些特殊类型

    除了上面提到的标量值类型，proto3中还定义了其他一些特殊的数据类型，方便我们用来构造、传递各种复杂的数据结构。官方给出了一个关于这些类型的JSON映射表，可以直观地看到各种类型所含数据的基本结构。

这些类型使得我们可以方便地构造出各种各样的数据形式。这其中有几个较为常用的类型，在小编进行的测试中经常遇到，下面我们就结合实际中的例子来为大家介绍一下。

    1、message

     message，根据映射表我们可以看到，它类似于我们在编程语言中所使用的类的对象（object）。在一个类中，我们可以添加各种其他类型的数据，也包括类本身。通过类比，message也有类似的概念，我们可以在里面添加各种proto类型的数据，也包括message。其实正如message的名字一样——消息，它是protobuf中的核心类型，在grpc接口中，我们正是通过发送和接收消息来完成数据交互，来实现对应的功能。

    简单的message：

message Person {   int32 id = 1;   string name = 2;   string email = 3;}

    含其他message的message：

message Point {    int32 latitude = 1;    int32 longitude = 2;}
message Feature {    string name = 1;    Point location = 2;}

    在Python中的使用：

location=Point(latitude=5,longitude=10)Feature=Feature(name="我是个名字",location=location)

    2、Timestamp、Duration

    这两种类型都是关于时间的，Timestamp是时间戳，Duration表示的时间长度。

    在AI平台账号服务的测试中，某Account类型的message定义如下：

message Account {  string account_id = 1;  google.protobuf.Timestamp update_at = 2;  google.protobuf.Duration time_limit = 3;}

    在Python中的使用：

update_at=Timestamp()#从字符串获取update_at.FromJsonString("1970-01-01T00:00:00Z")#获取当前时间update_at.GetCurrentTime()
time_limit=Duration()#从纳秒转换time_limit.FromNanoseconds(1999999999)#从秒转换time_limit.FromSeconds(100)
account=Account(account_id="account1",update_at=update_at,time_limit=time_limit)

    3、Any

    Any类型比较特殊，它可以包含不同的message，结合pack和unpack，只需声明一个Any，即可传递各种类型的message而不用声明多个字段。

    在大会同传项目中，某个请求的message中需要传递两种信息——图片和音频，于是通过Any类型来实现同一字段的复用：  

message ImageData {    string index = 1;    bytes  image = 2;}
message Data {    string appid = 1;    bytes payload = 2;    string extra = 3;}
message Request {    google.protobuf.Any body = 1;}

    在Python中的使用：

imageData=msg_pb2.ImageData(index="001",image=open("1.jpg","rb").read())req1=msg_pb2.Request()req1.body.Pack(imageData)
data=msg_pb2.Data(name="no.1",payload=open("1.wav","rb").read(),extra="no use")req=msg_pb2.Request()req.body.Pack(data)

    4、enum

    enum枚举类型和其他大多数编程语言的枚举类型概念相同，主要是通过提前设定好一些固定的值来限定可以传递的内容。

    在AI平台实名认证服务的测试中，需要一个认证人类型的字段，由于认证人类型收敛，于是使用enum类型来定义：

enum PersonType {  PERSONTYPE_UNSPECIFIED = 0;  INDIVIDUAL = 1;  LEGAL = 2;  AUTHORIZE = 3;}
message Person {  string real_name = 1;                 PersonType person_type = 2;}

    在Python中的应用：   

person_type=PersonType.Value("INDIVIDUAL")Person(real_name="小王",person_type=person_type)

    5、map

    map相当于json中的键值对，在Python中类似于字典（dict），我们可以利用Python的dict类型数据来对map进行设置。map在proto中声明时一般会带有尖括号，来指定key和value的具体类型，如map<string,string>就表示键值对的key、value都为string类型。

    在AI平台鉴权相关的测试中，需要为用户创建的应用绑定若干个不同的特殊属性，每个特殊属性对应着一个属性值，此处采用了map类型：

message App {  string appid = 1;  map<string, string> extra_informations = 2;}

    在Python中的应用：

extra_informations={"name":"app1","expired":"no"}app=App(appid="1234567", extra_informations=extra_informations)

    6、repeated

    repeated相当于json中的list，在Python中类似于列表（list），我们可以利用Python的list类型数据来对repeated进行设置。

    在AI平台账号服务的测试中，需要为账号添加各种不同的能力，每个能力有多个属性，而每个能力属性的种类和数据类型一致。此处采用了repeated类型：

message Audience {  string name = 1;  string tier = 2;}message Account {  string account_id = 1;  repeated Audience audience = 2;}

    在Python中的应用：

audience=[{"name":"ASR","tier":"stand"},{"name":"TTS","tier":"free"},{"name":"MT","tier":"stand"}]account=Account(account_id="account1",audience=audience)

三、实际应用中的问题与技巧

1、repeated类型赋值问题

    如果把上面所讲repeated类型例子中的Python代码改成如下形式，那么在运行时会报错：

audience=[{"name":"ASR","tier":"stand"},{"name":"TTS","tier":"free"},{"name":"MT","tier":"stand"}]account=Account(account_id="account1")account.audience=audience

错误信息：

AttributeError: Assignment not allowed to repeated field "name" in protocol message object.

    这与我们上面所说的message的两种赋值方式似乎有所出入，但事实是因为protobuf中的repeated类型并不是我们想象的那样与python中的list完全对应，因此在这里会出现问题。所以在实际应用中，我们应避免这种写法，尽量采用上面例子中的方式。另外我们还可以采用另外一种方式来达到同样的效果：

audience=[{"name":"ASR","tier":"stand"},{"name":"TTS","tier":"free"},{"name":"MT","tier":"stand"}]for audience1 in audience:    a=account.audience.add()    a.name=audience1['name']    a.tier=audience1['tier']

2、复杂message的数据构造问题

    在实际测试的接口中，有时某个message的结构可能会非常复杂，比如像语音识别服务一些接口，协议里包含很多不同的message和repeated类型，这样对于我们编写测试客户端代码以及构造case、解析case都会有一些影响。之前我们介绍过使用命令行的方式传递参数的方式显然难以满足这种情景下的需求，手动拼message的方式也显得十分不便。经过一番调研发现，对于这种情况，我们可以使用protobuf库中json_format里面的Parse、MessageToJson两个方法来有效解决，这两个方法可以实现protobuf message和json的互转。因为处理json的方式有很多，也很灵活，因此我们在构造case时可以使用json的方式，通过Parse方法直接将json转换成message。在收到返回结果之后，可以使用MessageToJson方法将message转换成json，这样对于我们测试人员来说，发送和接收的数据看起来都是json，无论是准备测试数据还检验结果都会轻松不少。

示例：

from google.protobuf import json_formatjson_obj='{"a1":1,"a2":2}'request = json_format.Parse(json_obj,MessageName())json_result = json_format.MessageToJson(request)print (json_result)

    其中MessageName为message的名称，json_result为转为json后的返回结果。

https://developers.google.com/protocol-buffers/

https://developers.google.com/protocol-buffers/docs/pythontutorial

Protocol Buffer Basics: Go  |  Protocol Buffers  |  Google Developers
https://developers.google.com/protocol-buffers/docs/gotutorial

一阶子探索：

1、python  --- protocol-buffers --- golang 

Protocol buffers are a language-neutral, platform-neutral extensible mechanism for serializing structured data.

Protocol Buffers - Google's data interchange format

Protocol Buffer Basics: Go  |  Protocol Buffers  |  Google Developers
https://developers.google.com/protocol-buffers/docs/gotutorial

Protocol Buffer Basics: Python  |  Protocol Buffers  |  Google Developers
https://developers.google.com/protocol-buffers/docs/pythontutorial

序列化和获取结构数据的方式

How do you serialize and retrieve structured data like this? There are a few ways to solve this problem:

• Use gobs to serialize Go data structures. This is a good solution in a Go-specific environment, but it doesn't work well if you need to share data with applications written for other platforms.  
• You can invent an ad-hoc way to encode the data items into a single string – such as encoding 4 ints as "12:3:-23:67". This is a simple and flexible approach, although it does require writing one-off encoding and parsing code, and the parsing imposes a small run-time cost. This works best for encoding very simple data.
• Serialize the data to XML. This approach can be very attractive since XML is (sort of) human readable and there are binding libraries for lots of languages. This can be a good choice if you want to share data with other applications/projects. However, XML is notoriously space intensive, and encoding/decoding it can impose a huge performance penalty on applications. Also, navigating an XML DOM tree is considerably more complicated than navigating simple fields in a class normally would be.

Protocol buffers are the flexible, efficient, automated solution to solve exactly this problem. With protocol buffers, you write a .proto description of the data structure you wish to store. From that, the protocol buffer compiler creates a class that implements automatic encoding and parsing of the protocol buffer data with an efficient binary format. The generated class provides getters and setters for the fields that make up a protocol buffer and takes care of the details of reading and writing the protocol buffer as a unit. Importantly, the protocol buffer format supports the idea of extending the format over time in such a way that the code can still read data encoded with the old format.

1/3、gobs -- golang pickling -- python   

https://golang.org/pkg/encoding/gob/

Package gob manages streams of gobs - binary values exchanged between an Encoder (transmitter) and a Decoder (receiver). A typical use is transporting arguments and results of remote procedure calls (RPCs) such as those provided by package "net/rpc".

The implementation compiles a custom codec for each data type in the stream and is most efficient when a single Encoder is used to transmit a stream of values, amortizing the cost of compilation.

跨语言性差，局限在golang

Use Python pickling. This is the default approach since it's built into the language, but it doesn't deal well with schema evolution, and also doesn't work very well if you need to share data with applications written in C++ or Java. 

11.1. pickle — Python object serialization — Python 2.7.16 documentation
https://docs.python.org/2/library/pickle.html

The pickle module implements a fundamental, but powerful algorithm for serializing and de-serializing a Python object structure. “Pickling” is the process whereby a Python object hierarchy is converted into a byte stream, and “unpickling” is the inverse operation, whereby a byte stream is converted back into an object hierarchy. Pickling (and unpickling) is alternatively known as “serialization”, “marshalling,” [1] or “flattening”, however, to avoid confusion, the terms used here are “pickling” and “unpickling”.

This documentation describes both the pickle module and the cPickle module.

11.1. pickle — Python object serialization — Python 2.7.16 documentation
https://docs.python.org/2/library/pickle.html#module-cPickle

pickle module. There are several differences, the most important being performance and subclassability.

First, cPickle can be up to 1000 times faster than pickle because the former is implemented in C. Second, in the cPickle module the callables Pickler() and Unpickler() are functions, not classes. This means that you cannot use them to derive custom pickling and unpickling subclasses. Most applications have no need for this functionality and should benefit from the greatly improved performance of the cPickle module.

The pickle data stream produced by pickle and cPickle are identical, so it is possible to use pickle and cPickle interchangeably with existing pickles. [10]

There are additional minor differences in API between cPickle and pickle, however for most applications, they are interchangeable. More documentation is provided in the pickle module documentation, which includes a list of the documented differences.

2/3、string

普通字符串的缺点是只能描述简单的数据结构

3/3、XML

跨语言性好，但是资源消耗高，性能差

更小更快

Developer Guide  |  Protocol Buffers  |  Google Developers
https://developers.google.com/protocol-buffers/docs/overview

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    required string number = 1;
    optional PhoneType type = 2 [default = HOME];
  }

  repeated PhoneNumber phone = 4;
}

Person person;
person.set_name("John Doe");
person.set_id(1234);
person.set_email("[email protected]");
fstream output("myfile", ios::out | ios::binary);
person.SerializeToOstream(&output);

fstream input("myfile", ios::in | ios::binary);
Person person;
person.ParseFromIstream(&input);
cout << "Name: " << person.name() << endl;
cout << "E-mail: " << person.email() << endl;

  <person>
    <name>John Doe</name>
    <email>[email protected]</email>
  </person>

# Textual representation of a protocol buffer.
# This is *not* the binary format used on the wire.
person {
  name: "John Doe"
  email: "[email protected]"
}

  cout << "Name: " << person.name() << endl;
  cout << "E-mail: " << person.email() << endl;

  cout << "Name: "
       << person.getElementsByTagName("name")->item(0)->innerText()
       << endl;
  cout << "E-mail: "
       << person.getElementsByTagName("email")->item(0)->innerText()
       << endl;

protoc -I=$SRC_DIR --python_out=$DST_DIR $SRC_DIR/addressbook.proto



https://mp.weixin.qq.com/s/eijIRBx-vKln0AftMqXkmQ