Changes in pod6 files not available after the first call to update-cache

[antoniogamiz]

You can see that happening in Raku/Documentable#73.

Maybe this question is relevant to the problem.

[JJ]

OK, the thing seems to be that as soon as you cal "load" it's cached, and it will not change.

[JJ]

The new version of 040-pod-extraction is failing now with:

Attempt to obtain non-existent POD for <a-pod-file>. Is the source new and failed to compile? Has the cache been updated?
  in method pod at /home/jmerelo/Code/forks/raku/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 214
  in block <unit> at t/040-pod-extraction.t line 34

This is due to the fact that, before, stuff was created and then tested in a different instance of a program. Now that we do it in the same instance it's not going to work. So that's the deeper problem I talk about in #17

[antoniogamiz]

I have been trying to unlock the cache as @JJ said here, without any success. I propose to use Pod::Load to serve the pods that have been modified. I mean, use Pod::Load in get-pods to serve those that have been modified and use the cache for those that have not.

[JJ]

The way I see it, there are several options

• Use a different mechanism to update the cache. Right now we are using high-level CompUnits, we should lower that level so that we don't turn the caches into something immutable.
• Use CompUnits at a higher level. Right now, CompUnits are a mechanism for retrieving precompiled code, which is version-based and has got its own indexing mechanisms. We are avoiding that mechanism altogether and using our own. Maybe we shouldn't; we should piggyback on it and add metainformation to current CompUnit caches.
• Hack the lock put on installed CompUnits and unlock it.

I think that the only reasonable purpose, relying on tested and documented APIs is the second one. The first is undocumented, and probably not frozen or specced. Ditto for third: we overcome some lock, it's bound to break. Besides, second would be also a way to re-use existing installations so that we can "install" documentation, for instance, and create HTML out of it, which would result in all kinds of good things.

[niner]

Can someone describe for me what you are actually trying to achieve and what kind of issues you are encountering? I'm a bit surprised that no one has ever talked to me about the best way to use our precompilation infrastructure. I could probably give you answers, so you don't have to guess and if there are no answers yet improve our API, so it covers your use case (whatever that is).

Also note that the stable API is defined by these classes and roles:

• CompUnit
• CompUnit::DependencySpecification
• CompUnit::Handle
• CompUnit::Loader
• CompUnit::PrecompilationRepository
• CompUnit::PrecompilationStore
• CompUnit::PrecompilationId
• CompUnit::PrecompilationDependency
• CompUnit::PrecompilationUnit
• CompUnit::Repository
• CompUnit::RepositoryRegistry
• Distribution
• Distribution::Resource
• Distribution::Resources

Everything else is subject to change. Everything not covered by spec tests is subject to change.

[JJ]

Thanks @niner, for dropping by. The basic idea is we want to use the precomp API for a POD cache. In a pinch I would describe it more or less like the current CUR infrastructure, but mutable. We would like to be able to cache a document we have already seen, and then check if the new version of the document is different (by date), and if it is, substitute it.

[niner]

Please don't assume that I know anything about our Pod stuff because....I don't :D
Why do you need a POD cache? Where and how should it be used? That's incidentally something that the README also doesn't answer.

Rakudo doesn't use timestamps for invalidating precomp files. The whole thing works only with checksums of file contents and for good reason. We learned the disadvantages of timestamps the hard way and I'd suggest you spare yourselves the pain and move on to checksums as well. After all the precomp infrastructure wants you to do that anyway.

I don't see why you'd have any issues with the precomp infrastructure for the use case as you've described it. After all, things work just fine when you develop a module in a FileSystem repository and make changes to source files. So I'm gonna need more information on your use case and your troubles.

[JJ]

El sáb., 29 feb. 2020 a las 8:52, niner (<notifications@github.com>) escribió:

Please don't assume that I know anything about our Pod stuff because....I don't :D Why do you need a POD cache? Where and how should it be used? That's incidentally something that the README also doesn't answer.

We need to generate the documentation site fast, so we need precompiled PODs for doing that. We'll open an issue to improve the README.md

Rakudo doesn't use timestamps for invalidating precomp files. The whole thing works only with checksums of file contents and for good reason. We learned the disadvantages of timestamps the hard way and I'd suggest you spare yourselves the pain and move on to checksums as well. After all the precomp infrastructure wants you to do that anyway.

OK. Any easy way of doing that?

I don't see why you'd have any issues with the precomp infrastructure for the use case as you've described it. After all, things work just fine when you develop a module in a FileSystem repository and make changes to source files. So I'm gonna need more information on your use case and your troubles.

Can you please tell us how to replicate the functionality you mention? We're not saying that it does not work, only we have no idea how to do that, which is why we're using the current mechanism, inspired by the code that was used in Pod::to::BigPage. So let's flesh out the use case * Initially, we'll cache (precompile) and index every file in the Raku/doc repo. Could be others, but that's the main use case. From those files, the primary and secondary HTML pages are generated. * When we update from the repo, some files will have changed. We need to know which are these; then we generate HTML pages only for those sites. There's a bit more here and there: generate graphs, things like that. But that's basically it. This issue is an example of what does not really work (and which were kind of mimicking so far, but now it does not, possibly after changes in 2020.02)