Feature, Add a lookup of Last-Modified in http file headers

[TheLazzziest]

Hi there!

Currently, I am working on a feature in dlt which leverages fsspec extensively. the library filesystem source lacks native support for standard HTTP endpoints. Many public data sources, such as those hosted on CDNs, are only accessible via HTTP, as demonstrated by this example: NYC trip data: Cloudfront.

In order to enable the support of public resources, dlt must know the last modification date of a resource which a CDN can provide in a standardized way, e.g.: RFC 1123. Thus, fsspec can use them to provide the date in turn.

[martindurant]

I am happy to include this. Please post a link to the RFC for posterity (I was not aware of it).

[TheLazzziest]

Hi @martindurant! Thank you for your quick feedback. I can, but it seems that the original document is quite confusing.

Don't you mind if I mention the document this way: RFC: 1123 -> RFC: 822#Section-5

It should become more obvious where the actual knowledge comes from.

In addition to that, I used this resource for a more distilled explanation. Let me know if that works.

[martindurant]

Thanks for the links, that will do nicely. I'll try to see why the CI test is reporting filenotfound for a URL that clearly does exist.

This is also my first time finding out about dlt, which I only briefly glanced at so far. It seems to have some ideas in common with intake.

[martindurant]

It turns out the failure was temporary.