"LOAD ALL THE THINGS" via CAML or DataStream

[DaleyKD]

I figured I'd continue the discussion here instead of on a closed PR (#452 ).

LoadDataAsStream is what the UI uses and it's optimized (via a ton of possible RenderOptions flags) to return a lot of information via a single call. Currently our parsing only handles the ListItem data, but I think this can be improved.

So seems you'd like to pull as much information about a given file or listitem in a single roundtrip...correct?

@jansenbe - The company I'm with (almost 10 years) does a LOT of SharePoint work. That's actually how I learned about SharePoint in 2011 (started with SharePoint 2010). Obviously as we moved to the App Model in 2013 and shortly after, SPO, things were a big change (going from feature activated to add-ins/CSOM).

The BIGGEST thing that has kicked our butts, specifically the last 5+ years is ... you guessed it... 429s.

We DO try to optimize our queries as much as possible, but the truth is, we usually need a LOT of stuff (e.g. - all the properties on a list item). We understand that's heavy on SharePoint, but there's not much we can do.

On top of that, we usually iterate all documents in a Doc Lib. LVT is a beast to work around, and customers who have a Doc Lib with a boat load of docs with a boat load of properties (fields/columns) tend to get our products throttled.

We believe that asking for as much data as possible in a single request helps with throttling vs. a bunch of requests.

Mind you, this was CSOM. Using PnP.Core in our current product is my first attempt using SharePoint REST, and with it, the $batch endpoint. Therefore, I'm not fully sure how requests are perceived by SharePoint using batches.

Is it better to do a batch with a LOT of requests, or is it better to do a batch with minimal heavy requests (aka - "give me 100 items at a time, with their property bag, content type ID, a few other things, and the file (oh, and include some of those properties, too)")?

I know that's long winded, but I hope it helps explain why I've been active in this repo and answers your question about the single roundtrip.

(For today, we're using the LoadDataAsStream endpoint, and then sending batches of x to web.GetFileByServerRelativeUrlBatchAsync. Then we match a list item to its file by ServerRelativeUrl.)

[jansenbe]

@DaleyKD :

Batching up the GetFileByServerRelativeUrl requests should reduce the 429's, PnP Core SDK will automatically split your batches if they're too big.

Below approach might however be a better solution (thanks for adding the extra properties 👍 ) for your needs, although it's not using LoadDataAsStream. Note that we do automatically page per 100 using this approach, so you would make a single call per 100 items in a library and request all the information you'd need for file/folder/listitem:

await foreach(var item in list.Items.QueryProperties(p => p.Id, p => p.Title, p => p.FileSystemObjectType,
                                                     p => p.ContentType.QueryProperties(c => c.Name, c => c.Id),
                                                     p => p.File.QueryProperties(f => f.Name, f => f.UniqueId),
                                                     p => p.Folder.QueryProperties(f => f.UniqueId, f => f.Name)).AsAsyncEnumerable())
{
    if (item.FileSystemObjectType == FileSystemObjectType.File)
    {
       // this is a file, use the item.File properties
    }
}

Since this uses an async enumerable in an async foreach the code inside the foreach loop can already run after the first page has been retrieved, possibly optimizing perf (or getting more 429's 😞). You can optionally specify a filter via the Where() method.

[jansenbe]

@DaleyKD : also, depending on how often your code is "crawling" SharePoint, you can use a model based upon web hooks and let SharePoint inform you of changes. See https://docs.microsoft.com/en-us/onedrive/developer/rest-api/concepts/scan-guidance?view=odsp-graph-online for more details