Indexed views

[srid]

Neuron's index is just a tree of links to all notes. This works on small sized notebooks, but can become intractable on really large ones (and can affect performance adversely; see #66).

One solution to this problem is to auto-collapse sub-trees known to have relatively large number of descendants, and delegate their listing to a dedicated child (or sub) index route. For instance, daily notes stored in "Calendar/" folder could have as much as 300 notes in a year, and if you are using the Zettelkasten for 5 years, that number reaches 1500. It makes no sense to list all daily notes in the index view. We could just automatically collapse the Calendar/ sub-tree in the main index, and then introduce the notion of "sub index" views.

If the index view has the route /-/index (which is what #49 implements), then any sub-index view for path P will have the route /-/index/$P. For the above Calendar/ folder, the sub-index view will be at /-/index/Calendar. And for nested sub-index views, such as say a specific month of a year, it will be at /-/index/Calendar/2020/07.

---

We can in fact generalize this "indexed" route-tree concept and apply it to hierarchical tags too. So a note tagged with foo/bar/qux will be listed in the /-/tags/foo/bar/qux tag index route, with "qux" itself listed in /-/tags/foo/bar (view demo) and so on.

---

• Delineated Tag Index Delineated Tag Index #85
  • Add backlink context to tag index pages (display the paragraph in which the inline tag is used) Tag-Index should show snippets of inline-tags. #208
• Delineated Folder Index (smart on-demand delineation?)
• Folgezettel Index (a la neuron's z-index)?
• Task Index #181

[srid]

Consolidate this with the sidebar tree configuration proposed in #64. See comment: #64 (reply in thread)

[wbadart]

I found this thread trying to track down what happened to the z-index in emanote. I definitely agree that neuron's z-index would not scale indefinitely, and that collapsible views could be part of the solution.

Thinking a little more aspirationally, maybe this is the opportunity to drop the tree view altogether and figure out some kind of graph visualization since back-links can form non-tree structures. We'd probably incur a frontend dependency to render the graph, but it gives way to a natural collapsing heuristic: hiding/ grouping nodes sharing parents of high out-degree.

[srid]

Thinking a little more aspirationally, maybe this is the opportunity to drop the tree view altogether and figure out some kind of graph visualization

Yes, this is in my mind. Requires #84 first though ...

[wbadart]

Glad to see you're already tracking this. Thanks for the link 👍

[srid]

I should clarify that the indexed view in its current form -- a static structure of links: be it neuron's folgezettel heterarchy dump, or a collapsible version being proposed here, or the full directory index of emanote in its current form -- is still useful, and should be retained. Dynamic graph view can be provided in addition to it. Only when the graph views satisfies all the use cases of indexed view may the latter be deprecated (otherwise, both should be kept).

I think any work on graph view would be in a very early research and exploration phase (I'm pretty new to visualization myself), so I don't know how it will turn out yet. The index view should be a good thing to have in the interim. I do know that I'm not a big fan of Obsidian's graph view, and as @j-steinbach notes below it is not a great index view (just that it looks pretty) ... but perhaps there is a way we can improve on that in an interactive fashion including #147 based interaction so as to provide a way to iteratively 'filter' the graph in real-time based on one's (changing) query view of what notes need to be displayed. All of this sounds like an independent project of its own, and maybe it should be explored in a separate repo first (will require #161).

[j-steinbach]

I believe the big question is: What exactly do we mean by index? Is it to display all notes? To display the important notes? To show an overview over collapsed categories? Also: What is its purpose? Look pretty? Provide a sense of pride and accomplishment by showing how big our notes folder is? Be useful? Let the user look up their notes-folder from the browser?

[Side-note: Our index is called "Impulse" -- "Full folgezettel heterarchy of the zettelkasten". What does that mean? Also we got a index.html page, which is user curated.]

If we assume our index to be intended as a listing of all notes, then we have the following problem: Our notes folder has no structure. We can't assume structure, and we mustn't enforce structure. It is just a loose pile of notes, spread over a few (possibly none) folders.

For example my Zettelkasten (ZK) consists of five folders:

• root :: ~2000 loose notes
• root/science :: ~1500 notes + bibliography
• root/literate :: my literate system config. ~100 files
• root/writings :: my novel + world-building ~100 files
• root/diary :: two years worth of daily notes (~700)

So how do we structure this?

• List all files :: This is what we currently do -- called "Full folgezettel heterarchy of the zettelkasten." Problem: A list of 2000 loose files doesn't help anyone.
• List all files, only show the parent nodes. :: This is what the visualization of the original ZK does. The index on the left only show the parent nodes, and we can explore the children. Problem: How do we deal with loose notes? Just list them under a category "Orphans"?
• List files based on their ids. Problem: This assumes Support content-dependant IDs #142
• List files alphabetically (Collapse them similar to Wikimedia subcategories) Problem: Internationalization. Example: What is the alphabetical order of CJK notes?
• Categorize based on tags. Similar to Wikimedia sidebar, under "Content". Problem: Is this still an index? Also we might have thousands of tags again (multi-lingual).

This means that a graph probably is the safest way to visualize the notes. But is a graph an index? In my opinion a graph is a visualization, which --whilst pretty and awesome-- doesn't have much "index" value:

The picture shows a cut-out of my Zettelkasten graph. The real one is 4x bigger and also contains notes without parents and links. I find it hard to identify small notes. The problem with graphs is that they change. In my opionion, an index should show me the most important parents/categories on a single glance, without changing every second day or requiring me to zoom around.

This does not mean that I am against graphs. Au contraire!, we definitely need a graph-based visualization. I just think that graphs and indices are different things.

[almereyda]

Your mockup reminds me of two things in particular:

• Gingko looks very similar to what you explore, and is easy to browse, scroll through with the arrow keys.
  
  • Gingko App
  • Gingko Writer
  • Gingko Inc. · GitHub
  • Releases · gingko/client · GitHub
• the Federated Wiki Title Network Browser uses a similar column layout to display two levels of each incoming (to the left) and outgoing (to the right) links.
  
  The numbers in this image visualise something very specific to the wiki federation:

  We sum the forks for each title and then show this and the title count as a column heading. Dividing these two numbers gives you the average degree of duplication.

[j-steinbach]

Yes, this is exactly what I meant. Simple to explore, high information-density and an elegant way to handle duplicate entries.

(Well, the Federated Wiki Title Network Browser however is a bit too dense on information for my tastes 😄 )

[dpaetzel]

I just wanted to ask whether by now there are any plans to incorporate indexed views á la neuron (or similar, some of the above proposals seem very promising as well) into emanote? Right now, this is the main reason that's holding me back from switching to emanote.

Anyway, thank you for your work, @srid ! 🙂

[srid]

@dpaetzel No plans so far, but I'm happy to guide others if they want to implement any of it. I can imagine two parts: adding extra route types (if necessary) and adding a 'render' function - which is a pure function taking 'Model' and returning HTML, effectively. Like, for tag index, we have:

• Route:

  emanote/src/Emanote/Route/SiteRoute/Type.hs

  Line 38 in 6f83b85

  newtype TagIndexR = TagIndexR [HT.TagNode]

• render:

  emanote/src/Emanote/View/TagIndex.hs

  Lines 79 to 80 in 6f83b85

  	renderTagIndex :: Model -> [HT.TagNode] -> LByteString
  	renderTagIndex model tagPath = do

[dpaetzel]

OK, thanks for pointing us in the general direction. When I find the time (though I'm not sure when that may be), I might give it a go.

[srid]

Note that Emanote now has JSON export; so these views can experimented with offline outside of Emanote before being upstreamed.

[srid]

I'm beginning to brainstorm over this. If anybody is interested, join the discussion over at Zulip.