Monty, Mongo tinified. MongoDB implemented in Python !

static security checking of codebase using Bandit revealed use of unsecure eval() function.

>> Issue: [B307:blacklist] Use of possibly insecure function - consider using safer ast.literal_eval.
   Severity: Medium   Confidence: High
   Location: montydb/types/_nobson.py:163
   More Info: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b307-eval
162                     if not _encoder.key_is_keyword:
163                         key = eval(candidate)
164                         if not isinstance(key, cls._string_types):

Have implemented ast.literal_eval() in its place

Dropping Python 3.4, 3.5 tests

In some test cases, for example:

• test/test_engine/test_find.py

  • test_find_2
  • test_find_3

• tests/test_engine/test_update/test_update.py

  • test_update_positional_filtered_near_conflict
  • test_update_positional_filtered_has_conflict_1

• tests/test_engine/test_update/test_update_pull.py

  • test_update_pull_6
  • test_update_pull_7

They often failed randomly due to the dict key order in run-time. I think, unless changing those test case documents into OrderedDict, or can not ensure the key order input into monty and mongo were the same ( which may cause different output ).

Since this is not the issue of montydb's functionality, dropping them for good.

Involving Python 3.7

Well, it's 2019.

MutableMapping should be imported from collections.abc as said in documentation https://docs.python.org/3.9/library/collections.abc.html#collections.abc.MutableMapping

Mainly we need it because its fix #65 (python3.10 compatibility)

Doesn't work with positional operators. Made the same update_one with pymongo successfully. Don't sure if monty got this feature yet not, but as I can see it causing error.

update_one({'users': {'$elemMatch': {'_id': id_}}}, {'$set': {'invoices.$.name': name}})

montydb.erorrs.WriteError: The positional operator did not find the match needed from the query.

Getting this error when making any operation after editing database. Using lmdb. Guessed it was after wrong update_one, but not sure about. Anyway, adding original code of editing db:

    record = {'free': 12313232, 'path': '/media/mnt/'}
    col = getattr(db, 'storages')

    record = {**models['storage'], **record}
    if col.count_documents({'path': record['path']}) > 0:
        col.update_one({'path': record['path']}, {'$set': {**record}})
    else:
        col.insert_one(record)

Error text:

Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/user/.local/lib/python3.8/site-packages/montydb/cursor.py", line 365, in next if len(self._data) or self._refresh(): File "/home/user/.local/lib/python3.8/site-packages/montydb/cursor.py", line 354, in _refresh self.__query() File "/home/user/.local/lib/python3.8/site-packages/montydb/cursor.py", line 311, in __query for doc in documents: File "/home/user/.local/lib/python3.8/site-packages/montydb/storage/lightning.py", line 253, in <genexpr> docs = (self._decode_doc(doc) for doc in self._conn.iter_docs()) File "/home/user/.local/lib/python3.8/site-packages/montydb/storage/__init__.py", line 227, in _decode_doc return bson.document_decode( File "/home/user/.local/lib/python3.8/site-packages/montydb/types/_bson.py", line 64, in document_decode return cls.BSON(doc).decode(codec_options) File "/home/user/.local/lib/python3.8/site-packages/bson/__init__.py", line 1258, in decode return decode(self, codec_options) File "/home/user/.local/lib/python3.8/site-packages/bson/__init__.py", line 970, in decode return _bson_to_dict(data, codec_options) bson.errors.InvalidBSON: objsize too large

That's how db looks like in plain:

$ cat db/storages.mdb @ @ ~60c9a9cf368c720edc2668a6{"path": "/mnt/hdd4", "total": 9999, "used": 99, "free": 9900, "status": "ready", "_id": {"$oid": "60c9a9cf368c720edc2668a6"}}~60c9a9cf368c720edc2668a5{"path": "/boot/efi", "total": 9999, "used": 99, "free": 9900, "status": "ready", "_id": {"$oid": "60c9a9cf368c720edc2668a5"}}y60c9a9cf368c720edc2668a4{"path": "/run", "total": 9999, "used": 99, "free": 9900, "status": "ready", "_id": {"$oid": "60c9a9cf368c720edc2668a4"}}v60c9a9cf368c720edc2668a3{"path": "/", "total": 9999, "used": 99, "free": 9900, "status": "ready", "_id": {"$oid": "60c9a9cf368c720edc2668a3"}} f*~s60c9a9cf368c720edc2668a3{"path": "/", "total": 9999, "used": 99, "free": 0, "status": "ready", "_id": {"$oid": "60c9a9cf368c720edc2668a3"}}2~60c9a9cf368c720edc2668a6{"path": "/mnt/hdd4", "total": 9999, "used": 99, "free": 9900, "status": "ready", "_id": {"$oid": "60c9a9cf368c720edc2668a6"}}~60c9a9cf368c720edc2668a5{"path": "/boot/efi", "total": 9999, "used": 99, "free": 9900, "status": "ready", "_id": {"$oid": "60c9a9cf368c720edc2668a5"}}y60c9a9cf368c720edc2668a4{"path": "/run", "total": 9999, "used": 99, "free": 9900, "status": "ready", "_id": {"$oid": "60c9a9cf368c720edc2668a4"} f*~sr60c9a9cf368c720edc2668a3{"path": "/", "total": 9999, "used": 99, "free": 0, "status": "busy", "_id": {"$oid": "60c9a9cf368c720edc2668a3"}}~60c9a9cf368c720edc2668a6{"path": "/mnt/hdd4", "total": 9999, "used": 99, "free": 9900, "status": "ready", "_id": {"$oid": "60c9a9cf368c720edc2668a6"}}~60c9a9cf368c720edc2668a5{"path": "/boot/efi", "total": 9999, "used": 99, "free": 9900, "status": "ready", "_id": {"$oid": "60c9a9cf368c720edc2668a5"}}y60c9a9cf368c720edc2668a4{"path": "/run", "total": 9999, "used": 99, "free": 9900, "status": "ready", "_id": {"$oid": "60c9a9cf368c720edc2668a4"}}

Or this:

$ cat db/storages.mdb @ @ 0 documents 0 document 0̝φħ^sTb_id̝φħ^sTpath/media/user/ssd1totalusedfreestatusreadyr60c9a9cf368c720edc2668a3{"path": "/", "total": 9999, "used": 99, "free": 0, "status": "busy", "_id": {"$oid": "60c9a9cf368c720edc2668a3"}}~60c9a9cf368c720edc2668a6{"path": "/mnt/hdd4", "total": 9999, "used": 99, "free": 9900, "status": "ready", "_id": {"$oid": "60c9a9cf368c720edc2668a6"}}~60c9a9cf368c720edc2668a5{"path": "/boot/efi", "total": 9999, "used": 99, "free": 9900, "status": "ready", "_id": {"$oid": "60c9a9cf368c720edc2668a5"}}y60c9a9cf368c720edc2668a4{"path": "/run", "total": 9999, "used": 99, "free": 9900, "status": "ready", "_id": {"$oid": "60c9a9cf368c720edc2668a4"}} 0 documents̝φħ^sTb_id̝φħ^sTpath/media/user/ssd1totalusedfreestatusreadyr60c9a9cf368c720edc2668a3{"path": "/", "total": 9999, "used": 99, "free": 0, "status": "busy", "_id": {"$oid": "60c9a9cf368c720edc2668a3"}}~60c9a9cf368c720edc2668a6{"path": "/mnt/hdd4", "total": 9999, "used": 99, "free": 9900, "status": "ready", "_id": {"$oid": "60c9a9cf368c720edc2668a6"}}~60c9a9cf368c720edc2668a5{"path": "/boot/efi", "total": 9999, "used": 99, "free": 9900, "status": "ready", "_id": {"$oid": "60c9a9cf368c720edc2668a5"}}y60c9a9cf368c720edc2668a4{"path": "/run", "total": 9999, "used": 99, "free": 9900, "status": "ready", "_id": {"$oid": "60c9a9cf368c720edc2668a4"} 0 0̝φħ^sTb_id̝φħ^sTpath/media/user/ssd1totalusedfreestatusreadyr60c9a9cf368c720edc2668a3{"path": "/", "total": 9999, "used": 99, "free": 0, "status": "busy", "_id": {"$oid": "60c9a9cf368c720edc2668a3"}}~60c9a9cf368c720edc2668a6{"path": "/mnt/hdd4", "total": 9999, "used": 99, "free": 9900, "status": "ready", "_id": {"$oid": "60c9a9cf368c720edc2668a6"}}~60c9a9cf368c720edc2668a5{"path": "/boot/efi", "total": 9999, "used": 99, "free": 9900, "status": "ready", "_id": {"$oid": "60c9a9cf368c720edc2668a5"}}y60c9a9cf368c720edc2668a4{"path": "/run", "total": 9999, "used": 99, "free": 9900, "status": "ready", "_id": {"$oid": "60c9a9cf368c720edc2668a4"}}

Does it stores changes after updating?

Performing updates with the flat file is giving me duplicate documents. I'm running Python 3.8.9 and have tried it with both the pip install montydb install and pip install montydb[bson] install. This problem is not occurring when in sqlite mode.

Subsequent program runs after inserting a record are causing a duplicate document to be added with the same _id.

It looks like the OrderedDict cache update at https://github.com/davidlatwe/montydb/blob/master/montydb/storage/flatfile.py#L79 is where the extra document is being added. Debugging the process shows that Python is adding a duplicate document because the keys in the ordered dict are actually different. One is an ObjectId object and the other is the binary serialized representation of that id. Here is a screenshot of the debugging output:

Here is the source code to reproduce. Note that you will have to run it twice because this only occurs on subsequent runs.

from montydb import MontyClient, set_storage

set_storage("./db/repo",  cache_modified=0)
client =  MontyClient("./db/repo")
coll = client.petsDB.pets

if  len([x for x in coll.find({"pet":  "cat"})])  ==  0:
    coll.insert_one({"pet":"cat",  "domestic?":True, "climate":  ["polar",  "equatorial",  "mountain"]})
    
coll.update_one({"pet":  "cat"},  {"$push":  {"climate":  "continental"}})
# This should only ever print 1 on subsequent runs.
print(len([x for x in coll.find({"pet":  "cat"})]))

Could not find by ObjectId. Code:

from bson.objectid import ObjectId

x = collection.find()
i = list(x)[0]['_id']

y = collection.find({'_id': ObjectId(i)})
print(list(y))

I see that database in mdb format has "_id": {"$oid": "id"} and I tried to find id in string with: {"_id.$oid": "string id"} but anyway, it would not work. Any suggestions? Thanks!

Issue

Cannot store bytes as values.

Env

Windows 10 Python 3.8.1 MontyDB 2.3.6

Actual error

>>> from montydb import MontyClient
>>> col = MontyClient(":memory:").db.test
>>> col.insert_one({'data': b'some bytes'})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\user\.virtualenvs\name\lib\site-packages\montydb\collection.py", line 139, in insert_one
    result = self._storage.write_one(self, document)
  File "C:\Users\user\.virtualenvs\name\lib\site-packages\montydb\storage\__init__.py", line 45, in delegate
    return getattr(delegator, attr)(*args, **kwargs)
  File "C:\Users\user\.virtualenvs\name\lib\site-packages\montydb\storage\memory.py", line 120, in write_one
    self._col[b_id] = self._encode_doc(doc, check_keys)
  File "C:\Users\user\.virtualenvs\name\lib\site-packages\montydb\storage\__init__.py", line 183, in _encode_doc
    return bson.document_encode(
  File "C:\Users\user\.virtualenvs\name\lib\site-packages\montydb\types\_bson.py", line 236, in document_encode
    for s in _encoder.iterencode(doc):
  File "C:\Program Files\Python38\lib\json\encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "C:\Program Files\Python38\lib\json\encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "C:\Program Files\Python38\lib\json\encoder.py", line 438, in _iterencode
    o = _default(o)
  File "C:\Users\user\.virtualenvs\name\lib\site-packages\montydb\types\_bson.py", line 222, in default
    return NoBSON.JSONEncoder.default(self, obj)
  File "C:\Program Files\Python38\lib\json\encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable

Same with PyMongo

>>> from pymongo import MongoClient
>>> col = MongoClient('127.0.0.1').tests.test1
>>> col.insert_one({'data': b'some bytes'})
<pymongo.results.InsertOneResult object at 0x000002BBB52BC7C0>
>>> next(col.find())
{'_id': ObjectId('60bdaa528ff3727b58f514f7'), 'data': b'some bytes'}

Instead of selecting a handful of vital tests to run, let’s run all flake8 tests ignoring only a handful of tests.

flake8 . --ignore=E302,F401,F841,W605

Hello guys, I find your project very interesting but there has not been any development for quite a while. Is this project not under development anymore? Kind regards

Hi @davidlatwe, Just discovered that monty(montydb-2.3.10) can't handle query like:

x = col.find({"mapping": {'$elemMatch': {'$elemMatch': {'$in': [https://accounts.google.com/o/oauth2/aut']}}}})

for a array in array elements:

"mapping": [ ["https://accounts.google.com/o/oauth2/auth", "client_id", "redirect_uri", "scope", "response_type"] ]

Just doesn't find anything, unlike pymongo does

Updating a document in an array leads to an error: https://docs.mongodb.com/manual/reference/operator/update/positional/#update-documents-in-an-array

  collection.update_one(
      filter={
          "order": order_number,
          "products.product_id": product_id,
      },
      update={
          "$set": {
              "products.$.quantity": quantity
          }
      }
  )

leads to an exception:

            else:
                # Replace "$" into matched array element index
                matched = fieldwalker.get_matched()
>               position = matched.split(".")[0]
E               AttributeError: 'NoneType' object has no attribute 'split'
\field_walker.py:586: AttributeError

When using the $slice projection together with an exclusion projection, the operation should return all the other fields in the document. https://docs.mongodb.com/manual/reference/operator/projection/slice/#behavior

Monty DB is only returning the array that is sliced.

NotImplementedError: 'MontyCollection.aggregate' is NOT implemented ! It would be awesome to have aggregate, this project is very cool non the less!

Montydb with the sqlite backend provides multi-process operation, at least in my initial trials with just 2 processes writing to the database simultaneously. This is clearly an advantage over mongita, which also provides a file/mem clone of pyMongodb; however, doesn't provide multi-process support. But mongitadb does support Mongoengine which was achieved recently.

Has anyone been able to use Montydb with Mongoengine?