jeudi 13 août 2015

Python compare dictionary to dictionary of conditions

I have an iterator (millions of rows) that gives me a dictionary that need to be compared against a dictionary of conditions to find matches.

Here is my code:

conditions={"port":"0-20", "ip":"1.2.3.4", "protocol":"1,7",
            "timestamp":">143990000", "server":"mario"}

for rec in imiterator(): # Very large number of rows
    # rec examples {"ip":"1.7.1.1", "timestamp":1434000,
    #              "port":129,"server":("mario","bruno"), 
    #              "protocol":"1","port":19"}

    if check_conditions(rec, conditions):
       print(json.dumps(rec))

Note the columns in rec can be int, long, string, tuple.

I need to find a real high performance way to do the matches. Any ideas?

I thought about using map and converting the conditions to lambda functions that should match and doing an AND operation of all the conditions. Would this be faster?



via Chebli Mohamed

Aucun commentaire:

Enregistrer un commentaire