pip install hdfs
python read hdfs directory or file
import hdfs
client =hdfs.Client("http://10.10.1.4:50070")
fileDir="/user/hive/warehouse/house.db/dm_house/dt=201800909"try:
status=client.status(fileDir,False)if status:print(status)
rst=client.download(fileDir,"/home/dev/gewei")print(rst)
exception Exception as e:print(e)
Supplementary knowledge: Webhdfs cannot be found when accessing hdfs with python
It may be that the webhdfs service is not turned on
Add attributes to the hdfs-site.xml file:
< property
< name dfs.webhdfs.enabled</name
< value true</value
< /property
You can use the following command to detect,
Get a list of directories:
curl -i “http://Hadoop:50070/webhdfs/v1/?user.name=hadoop&op=LISTSTATUS”
The above operation of python access hdfs is all the content shared by the editor, I hope to give you a reference.
Recommended Posts