ref does not return all tickers all of the time

[jaksiprejak]

When I repeatedly call for a fields on a list of tickers with ref, I occasionally get back an incomplete dataframe i.e. values for some tickers did not get returned.

Code sample and debug output below.

I am running with my session connected to local terminal, and with pdblp version 0.1.8

import pdblp

con = pdblp.BCon(debug=True, timeout=1500)
con.start()

tickers = [
    f'EUSA{i}' for i in range(1, 31) 
]

fields = ['BID', 'ASK']

while True:
    print('##################')
    try:
        df = con.ref([f'{t} Curncy' for t in tickers], fields)
        if df.shape != (60, 3):
            print(df.to_string())
            break
    except Exception as e:
        print(e)  # to catch timeout

Debug output

##################
pdblp.pdblp:INFO:Sending Request:
ReferenceDataRequest = {
    securities[] = {
        "EUSA1 Curncy", "EUSA2 Curncy", "EUSA3 Curncy", "EUSA4 Curncy", "EUSA5 Curncy", "EUSA6 Curncy", "EUSA7 Curncy", "EUSA8 Curncy", "EUSA9 Curncy", "EUSA10 Curncy", "EUSA11 Curncy", "EUSA12 Curncy", "EUSA13 Curncy", "EUSA14 Curncy", "EUSA15 Curncy", "EUSA16 Curncy", "EUSA17 Curncy", "EUSA18 Curncy", "EUSA19 Curncy", "EUSA20 Curncy", "EUSA21 Curncy", "EUSA22 Curncy", "EUSA23 Curncy", "EUSA24 Curncy", "EUSA25 Curncy", "EUSA26 Curncy", "EUSA27 Curncy", "EUSA28 Curncy", "EUSA29 Curncy", "EUSA30 Curncy"
    }
    fields[] = {
        "BID", "ASK"
    }
    overrides[] = {
    }
}

pdblp.pdblp:INFO:Event Type: 'RESPONSE'
pdblp.pdblp:INFO:Message Received:
ReferenceDataResponse = {
    securityData[] = {
        securityData = {
            security = "EUSA1 Curncy"
            eidData[] = {
            }
            fieldExceptions[] = {
            }
            sequenceNumber = 0
            fieldData = {
                BID = -0.265300
                ASK = -0.252700
            }
        }
        securityData = {
            security = "EUSA2 Curncy"
            eidData[] = {
            }
            fieldExceptions[] = {
            }
            sequenceNumber = 1
            fieldData = {
                BID = -0.325200
                ASK = -0.311000
            }
        }
        securityData = {
            security = "EUSA3 Curncy"
            eidData[] = {
            }
            fieldExceptions[] = {
            }
            sequenceNumber = 2
            fieldData = {
                BID = -0.334100
                ASK = -0.318400
            }
        }
        securityData = {
            security = "EUSA4 Curncy"
            eidData[] = {
            }
            fieldExceptions[] = {
            }
            sequenceNumber = 3
            fieldData = {
                BID = -0.319000
                ASK = -0.307000
            }
        }
        securityData = {
            security = "EUSA5 Curncy"
            eidData[] = {
            }
            fieldExceptions[] = {
            }
            sequenceNumber = 4
            fieldData = {
                BID = -0.294200
                ASK = -0.283900
            }
        }
        securityData = {
            security = "EUSA6 Curncy"
            eidData[] = {
            }
            fieldExceptions[] = {
            }
            sequenceNumber = 5
            fieldData = {
                BID = -0.267400
                ASK = -0.254000
            }
        }
        securityData = {
            security = "EUSA7 Curncy"
            eidData[] = {
            }
            fieldExceptions[] = {
            }
            sequenceNumber = 6
            fieldData = {
                BID = -0.233800
                ASK = -0.222400
            }
        }
        securityData = {
            security = "EUSA8 Curncy"
            eidData[] = {
            }
            fieldExceptions[] = {
            }
            sequenceNumber = 7
            fieldData = {
                BID = -0.197300
                ASK = -0.186500
            }
        }
        securityData = {
            security = "EUSA9 Curncy"
            eidData[] = {
            }
            fieldExceptions[] = {
            }
            sequenceNumber = 8
            fieldData = {
                BID = -0.160800
                ASK = -0.148800
            }
        }
        securityData = {
            security = "EUSA10 Curncy"
            eidData[] = {
            }
            fieldExceptions[] = {
            }
            sequenceNumber = 9
            fieldData = {
                BID = -0.122700
                ASK = -0.110400
            }
        }
    }
}

           ticker field   value
0    EUSA1 Curncy   BID -0.2653
1    EUSA1 Curncy   ASK -0.2527
2    EUSA2 Curncy   BID -0.3252
3    EUSA2 Curncy   ASK -0.3110
4    EUSA3 Curncy   BID -0.3341
5    EUSA3 Curncy   ASK -0.3184
6    EUSA4 Curncy   BID -0.3190
7    EUSA4 Curncy   ASK -0.3070
8    EUSA5 Curncy   BID -0.2942
9    EUSA5 Curncy   ASK -0.2839
10   EUSA6 Curncy   BID -0.2674
11   EUSA6 Curncy   ASK -0.2540
12   EUSA7 Curncy   BID -0.2338
13   EUSA7 Curncy   ASK -0.2224
14   EUSA8 Curncy   BID -0.1973
15   EUSA8 Curncy   ASK -0.1865
16   EUSA9 Curncy   BID -0.1608
17   EUSA9 Curncy   ASK -0.1488
18  EUSA10 Curncy   BID -0.1227
19  EUSA10 Curncy   ASK -0.1104

[matthewgilbert]

This seems odd. Are you able to reproduce this issue when you make the call directly via blpapi? Does this issue always happen or is it intermittent?

[jaksiprejak]

I haven't had a chance to code it up with the blpapi, however I ripped out the bbg stuff from tia and ran 2to3 on it. It's been running for over an hour now with no missing ticker issues.

Have you been able to reproduce?

I'm wondering if it is something to do with the timeouts. I get no timeout errors with the tia code (I assume they're handling them internally).

I have been experiencing an odd issue with multithreading and timeouts as well. I have a bunch of threads with their own queues (they're actually actors from pykka). Each one has it's own BCon object. They're all basically calling ref on with arguments they get from their queues. They then return the resultant DataFrames with the arguments used for the request.
The ref() calls are wrapped in a while...try...except retry loop to handle timeouts. When a timeout occurs, the next successful ref() seemingly returns data from a request that one of the other threads made - evidenced by the fact that a different set of tickers is returned than the ones requested.

I haven't been able to boil this down to a simple code example to reproduce - it could very well be in my logic (although after a few hours I don't see the problem). I just thought I would mention it in case it would trigger something in your mind to help identify the original issue.

[jaksiprejak]

Does this issue always happen or is it intermittent?

The output I posted is from the final iteration of the while loop. It ran successfully 104 times before that.
However I am running a lot of ref calls in parallel so am seeing it fairly often.

[matthewgilbert]

the next successful ref() seemingly returns data from a request that one of the other threads made - evidenced by the fact that a different set of tickers is returned than the ones requested.

Hmm, okay from the sounds of it possibly the default blpapi.EventQueue for the blpapi.Session is getting populated with multiple requests. This shouldn't happen across the BCon instances but for each individual instance there could be some timeout/ backlog issue.

This library doesn't use a seperate blpapi.EventQueue() for each request so this type of in parallel requesting could be problematic and is a use case I haven't really used / tested for. It would definitely make sense to change the blpapi.Session.sendRequest calls to use their own blpapi.EventQueues but I don't think I will get around to making that modification any time soon.