Mongodb and python interaction of python crawler

mongodb and python interaction##

learning target#####

Master the method of adding, deleting, modifying and checking the interaction between mongdb and python
Use the pymongo module to master the authorization authentication method

1. Mongdb and python interaction module###

pymongo provides all the methods of interaction between mongdb and python
Installation method: pip install pymongo

2. Use pymongo

2.1 Import pymongo and select the collection to operate####

Database and collection can be created automatically

2.1.1 Create connection objects and collection operation objects without authorization authentication#####

from pymongo import MongoClient

client =MongoClient(host,port) #If it is a local connection host,The port parameter can be omitted

collection = client[db name][Collection name]
# collection = client.db name.Collection name#Same usage as above

2.1.2 Create connection objects and collection operation objects in a way that requires authorization

from pymongo import MongoClient
from urllib.parse import quote_plus

user ='python' #account number
password ='python' #password
host ='127.0.0.1' # host
port =27017 # port
uri ="mongodb://%s:%s@%s"%(quote_plus(user),quote_plus(password),
        host)
# quote_plus function: encode the url
# uri = mongodb://python:[email protected]
client =MongoClient(uri, port=port)
collection = client.db name.Collection name

2.2 insert()Add data####

insert can insert a list of data in batches, or insert a piece of data

collection.insert({A piece of data})
collection.insert([{Data One},{Data two}])

2.2.1 Add a piece of data

Return the _id of the inserted data

ret = collection.insert({"name":"test10010","age":33})print(ret)

2.2.2 Add multiple data

Returns a list of ObjectId objects

item_list =[{"name":"test1000{}".format(i)}for i inrange(10)]
rets = collection.insert(item_list)print(rets)for ret in rets:print(ret)

2.3 find_one()Find a piece of data####

Receive a condition in dictionary form and return the entire data in dictionary form
If the condition is empty, return the first one

ret = client.test.test.find_one({'name':'test10001'})print(ret) #A dictionary containing the ObjectId objects of mongodb
_ = ret.pop('_id') #Clear the k of the ObjectId object of mongodb,v
print(ret)

2.4 find()Find all data####

Return all results that meet the condition, if the condition is empty, return all
The result is a Cursor cursor object, an iterable object, which can be similar to reading a file pointer, but it can only be read once

rets = collection.find({"name":"test10005"})，
for ret in rets:print(ret)for ret in rets: #There is no content in rets at this time
 print(ret)

2.5 update()Update data (overwrite the entire document or specify the key value, update one or more items)

Syntax: collection.update({condition}, {'$set':{specified kv or a complete piece of data}}, multi=False/True, upsert=False/True)
multi parameter: the default is False, which means to update one; multi=True to update multiple; multi parameter must be used with $set
upsert parameter: the default is False; upsert=True first query whether it exists, if it exists, update; if it does not exist, insert
$set means the specified field is updated

2.5.1 Update a piece of data; full document coverage; update if it exists, insert if it doesn't exist

data ={'msg':'This is a complete piece of data 1','name':'Haha'}
client.test.test.update({'haha':'heihei'},{'$set':data}, upsert=True)

2.5.2 Update multiple pieces of data; full document coverage; update if it exists, insert if it does not exist

data ={'msg':'This is a complete piece of data 2','name':'Haha'} #The complete data is obtained after querying
client.test.test.update({},{'$set':data}, multi=True, upsert=True)

2.5.3 Update a piece of data; specify the key value; update if it exists, insert if it does not exist#####

data ={'msg':'Specify to update msg only___1'}
client.test.test.update({},{'$set':data}, upsert=True)

2.5.4 Update multiple pieces of data; specify the key value; update if it exists, insert if it does not exist

data ={'msg':'Specify to update msg only___2'}
client.test.test.update({},{'$set':data}, multi=True, upsert=True)

2.6 delete_one()Delete a piece of data####

collection.delete_one({"name":"test10010"})

2.7 delete_many()Delete all data####

collection.delete_many({"name":"test10010"})

3. pymongo module other api

Check pymongo official documentation or source code http://api.mongodb.com/python/current/

summary##

Master the use of pymongo's addition, deletion and modification
Use the pymongo module to master the authorization authentication method