bug说明
客户端向web服务器发请求时,web服务器会分块返回数据,在每一块的第一行指明该块的长度(16进制)
参见 HTTP协议的Transfer-Encoding报头
一般web服务器,将所有块都发送完毕时,会发送0,表示没有数据了,如下(第17行的0):
HTTP/1.1 200 OK
Content-Type: text/plain
Transfer-Encoding: chunked
25\r\n
This is the data in the first chunk\r\n
1C\r\n
and this is the second one\r\n
3\r\n
con\r\n
8\r\n
sequence\r\n
0\r\n
\r\n
但有一些网站,如https://www.oschina.net/ (Tengine服务器),当没有块时,发送空字节,此时http.client模块会报错,代码如下:
import http.client
domain='www.oschina.net'
url='/'
try:
conn=http.client.HTTPSConnection(domain)
conn.request("GET",url)
resp = conn.getresponse()
print(resp.status, resp.reason,resp.version)
print(resp.headers)
if resp.status==200:
data = resp.read()
contenttype=resp.getheader('Content-Type')
if contenttype.find('charset=')>=0:
charset=contenttype.split('charset=')[1]
else:
charset='utf-8'
print( data.decode(charset))
except :
conn.close()
raise
错误如下:
File "c:/Users/mym/Desktop/python/http/client.py", line 572, in _readall_chunked
raise IncompleteRead(b''.join(value))
__main__.IncompleteRead: IncompleteRead(81517 bytes read)
bug解决
报错的原因是http.client模块没有判断空行,解决bug方法如下:
在http.client.py文件的_read_next_chunk_size函数(大约在506行),修改如下,增加第4、第5行代码:
def _read_next_chunk_size(self):
# Read the next chunk size from the file
line = self.fp.readline(_MAXLINE + 1)
if line==b'': #判断当为空时就返回0
return 0
if len(line) > _MAXLINE:
raise LineTooLong("chunk size")
i = line.find(b";")
if i >= 0:
line = line[:i] # strip chunk-extensions
try:
return int(line, 16)
except ValueError:
# close the connection as protocol synchronisation is
# probably lost
self._close_conn()
raise