Assessment of Transformative Programs

[rufuspollock]

We're interested in assessing transformative programs including our own ECT. We need to identify both factors we wish to assess and measures for assessing them.

👉 Associated note 📓: https://notes.lifeitself.org/assessment-of-transformative-programs

Existing Work

• Inner Development Goals
• Wisdom related measures

Examples of assessment scales used by Martin and NonSymbolic

From Martin, J. A. (20210909). Effects of two online positive psychology and meditation programs on persistent self-transcendence. Psychology of Consciousness: Theory, Research, and Practice https://doi.org/10.1037/cns0000286 (in library)

Participants were assessed using the Authentic Happiness Inventory, Satisfaction with Life Scale, PERMA, Fordyce Emotions Questionnaire, Center for Epidemiology Studies Depression questionnaire, State/Trait Anxiety Inventory, Perceived Stress Scale, Gratitude Questionnaire, Mysticism Scale, Modified Nondual Embodiment Thematic Inventory, and Meaning in Life Questionnaire. After the program, participants were sorted into seven categories of self-transcendence: none, temporary, and five increasing degrees of persistent self-transcendence. Results from each measure were broken out by category, and compared within and across cohorts.

NB: this paper is a nice example of rigorously measuring and assessing a novel and related area

[rufuspollock]

Added some suggestions to doc re https://www.authentichappiness.sas.upenn.edu/testcenter

[baouroux]

Fascinating thread and reference to the research paper...I'm very interested in the 1st, 2nd and 3rd person aspects in such program designs.

• 1st person including the contemplative practices extending the net of potential realms of experience to the nature-immersion (emulating the vision-quest, Joseph Jarworski Source type inspirations) and artistic expression / creative symbolic acts.
• 2nd person in terms of the mutuality of transformative programs, the witnessing, the holding, the being seen by others
• 3rd person in terms of the constructive-developmental (Robert Kegan) meaning-making shifts before/after the program (looking at the dynamic vs. any 'stage')

The above is my making sense of own explorations over the years, from attending the Mankind Project, to retreats etc.!

[rufuspollock]

@baouroux this is a really useful classification and how we could use that to structure both analysis of evaluation methods and to classify programs or techniques as well.

[rufuspollock]

UPDATE: posted this to the notebook/wiki at https://notes.lifeitself.org/notes/Pathways to Liberation Matrix

Adding Pathways to Liberation matrix from the NVC world: https://pathwaystoliberation.com/the-matrix/

• Spreadsheet version (latest): https://docs.google.com/spreadsheets/d/1KPIvCD_XIGThKEsqZ-qKQwY0Op5pSe_r7uqksetW5FQ/edit#gid=10
• Online self-assessment with backing database: https://pathwaystoliberation.com/the-matrix-instructions/
• Pathways-to-Liberation-Self-Assessment-Matrix.pdf

Comments:

• Very solid set of characteristics
• Big bonus is that it is openly licensed under Creative Commons attribution license.

(h/t to Karl Steyaert for flagging this to me).

Full list of qualities and their short descriptions

• Presence: Being attentive to what is happening right now. Not lost in thinking, emotional reactions, etc.
• Observing: Noticing (and possibly describing) our sensory and mental experiences, and distinguishing these experiences from the interpretations we ascribe to them.
• Feelings Awareness: Ability to identify and experience our physical sensations and emotions.
• Self-acceptance: Accepting oneself with unconditional caring.
• Taking ownership of one's feelings: Living from the knowledge that I alone cause my emotions - my emotions are not caused by others.
• Needs consciousness: Awareness of (and the willingness to honor) needs, the essential universal elemental qualities of life (like sustenance, love and meaning).
• Reconnecting to self and recovering from reactivity: Reactivity is internal resistance to what is. Recovery is letting go of that resistance. Re-connecting to self is being with one's own experience with presence and compassion.
• Request consciousness & making requests: Willingness to ask for what one wants, with openness to any response; not attached to any particular outcome.
• Mourning: Transforming the suffering of loss; letting go of resistance to what is, and being willing to allow our experience to unfold.
• Empathy: Being present with another's experience, with unconditional acceptance of the person.
• Dissolving enemy images: Transcending one's perceptions that another deserves to be punished or harmed.
• Discernment: Clarity, insight, and wisdom in making life-serving distinctions and choices; recognizing one has choice.
• Living interdependently: Living from the knowledge that every individual is related to every other individual - every part of a system affects every other part.
• Honest self-expression: Owning one's experience and having the willingness to express authentically without blame or criticism.
• Facilitating connection: Facilitating empathy and honesty in dialogue with an intent to create connection.
• Patience: Remaining spaciously present when one feels stress. An ability to be with one's own reactions, without acting out of them.
• Responding to others' reactivity: Responding rather than reacting to others who are caught up in intense separating emotions.
• Openness to feedback: Receiving other's perspective about our actions with equanimity and centeredness.
• Beneficial regret: Acknowledging and learning from one's missed opportunity to meet needs, without guilt, shame, or self-punishment.
• Flexibility in relating: Openness and versatility in interacting with others.
• Transforming conflict: Using conflict with others as a means to connect and create a mutual outcome.
• Gratitude: Finding the value in, appreciating, and enjoying what is.
• Open-hearted flow of giving and receiving: Transforming scarcity thinking into thriving creatively; joyfully contributing and receiving.
• Cultivating vitality: Tuning in to oneself to support balanced self-care; cultivating the energy to serve life.
• Sharing power: Transforming domination; valuing everyone's needs with mutuality and respect; transcending submission and rebellion.
• Transcending roles: Aware that we are not the roles we play; having choice about what roles we adopt and how we respond to the roles others adopt.
• Awareness of response-ability: Freely choosing one's responses to what shows up in life, owning one's part in what happens. Not owning others' parts, and acknowledging that one's actions do influence others.
• Supporting holistic systems: Consciously participating in the creation and evolution of holistic systems that foster general well-being.

[asimong]

not sure what you mean by “full” @rufuspollock ? Oh, sorry, my oversight here, you mean full in the sense of all the ones listed by Pathways to Liberation

[asimong]

I have a real problem with this matrix, not as a concept, which seems fair enough, but as a self-assessment matrix. Most of the descriptors are clearly value laden. The ones on the right are clearly positively valenced and more desirable. The ones on the left have various derogatory words within the description, such that I wonder whether people at those levels would actually self-identify with them.

To my mind, the essence of an effective self-assessment grid like this is to present each cell as something that people can identify with. I've made one attempt to do this (I'm sure it can be improved on!) for emotional self-connection here: https://wiki.simongrant.org/doku.php/ch:cr:emotional_self-connection

If anyone knows of anyone (themselves or others!) who would be willing to transform this matrix into something that would pass the test of people being able to identify positively with the descriptors, using non-judgemental words (hey, NVC!) then I would be delighted to collaborate.

[rufuspollock]

Excellent points @asimong

There are two parts here: identifying useful distinct qualities and then finding a way to have a framework for assessment of those (that works well - "well" needing to be defined).

[asimong]

Thanks Rufus @rufuspollock — well then, is anyone up for having a go at re-interpreting them for self-assessment use?

[asimong]

actually, I suspect that the process of working out a value neutral self assessment scale will actually help to understand how relevant the skill itself is

[Liyubov]

For assessment of programs the big need is indeed to collect some materials for assessment, as discussed during the talk by @baouroux .
John, if you have some links on best practices to share on

• how to collect the "warm data" or "unstructured data" from narratives
• how to make surveys for such programs

Thank you!

[Liyubov]

Some good practices for self-research documentation practices with Wikis from OpenHumans community
https://wiki.openhumans.org/wiki/Help:How_to_share_projects
Good to keep in mind @msantolini

[msantolini]

Thanks @Liyubov, I indeed wanted to point out to Open Humans in the context of citizen science of contemplation, as I believe they would provide an ideal platform to frame the project.

[baouroux]

Hi Liubov, Thanks for the message on the 6th, taking a little time to respond to you on the possible narratives/Sensemaking lens for assessment of transformative programs. Herewith a few references - in putting them together, I couldn't help weave in something of a design manifesto for the overall design choices, which follows as a second section below. - *Collecting Warm Data and Surveys*: - *Asymmetric/Remote Gathering:* The classic approaches are the survey platform with open narratives prompts, and self-signification trade-off sliders (either 2 point slider or 3 polarities triad) - A good overview of the platforms available, with comparison criteria are: https://pni2.org/2015/11/comparison-of-narrafirma-with-other-platforms-for-pnisensemaking/ - To add to the list in the above table is www.spryng.io, which has some good reference material there too. - My previous consulting brand developed a number of software modules for data visualisation in complement to SenseMaker <https://thecynefin.co/how-to-use-sensemaker/> from Cynefin.io, but I sold that on, and am now relaunching as more of a 'dialogue' platform vs. narratives assessment platform (www.collectivenavigator.com). - Cynefin have a good outline of the narratives method here: https://cynefin.io/wiki/SenseMaker - Lots more narratives based case study materials available that I can send you, as a follow-up. - - *Live Gathering*: There are then a range of story sharing, story circles practices etc that are typically live facilitated - Including integrated with World Café / Liberating Structures type events, where you can capture on paper / meeting room boards / audio / survey platform entry. - Cynefin used to have a live events software product called MassSense, but only now referred to in their methods description wiki here: https://cynefin.io/wiki/MassSense *Some Complexity Design Principles And Reflections For Assessment of Transformation:* - *Construct Aware:* What is the underlying construct of our capture design? For example, is personal transformation something that can be measured through language based surveys, designed from a set of theoretical frameworks? Reflections on how transformation experiences might fall into the Cynefin "Complex" and even "Chaotic" domain: - The assessment in the Complex domains for example would be more orientated towards the quality of the experience of the process ("describe your experience of the process"; "what stories arose for you during the experience?"), than measuring the impact/outcomes ("how did the experience transform you, across this range of developmental indicators?"). - *Methodological Pluralism*: How might personal transformation best be observed through multiple domains, such as the mental, physical, emotional, interpersonal, being (including creative, precognitive experiences) (gross/subtle/causal) / across the 1st person (self reflection), 2nd person (peer feedback) and 3rd person (physiological etc.). - E.g. What shows up differently between a survey question, a creative expression (working with metaphor), an improvisation, a conversation? - *Obliquity:* Within the complexity sciences principles, "obliquity" is an important one in capturing "sensemaking narratives" (as referred to in this book, Obliquity: Why Our Goals Are Best Achieved Indirectly" by John Kay), where direct, closed questions are avoided - *Engagement Currency*: Design consideration is that the experience / feedback / story sharing process has to have "something in it" for the participant. - How can the design of the transformation assessment be integrated into the transformative experience itself - e.g. intensive journaling process (Ira Progoff), combined with self-signification) - *Bottom-Up / Interactive Signification*: Shifting from a "framework or theory led assessment" approach to a "practice-led theory" where to the signification is bottom-up rather than top-down: - Repertory grid - George Kelly self sensemaking analysis https://en.wikipedia.org/wiki/Repertory_grid - The repertory grid is a technique for identifying the ways that a person construes (interprets or gives meaning to) his or her experience. It provides information from which inferences about personality can be made, but it is not a personality test in the conventional sense. It is underpinned by the personal construct theory developed by George Kelly, first published in 1955. - There are some interactive / algorithm / AI avenues I can see with the dynamic signification from developmental / transformational libraries (as mentioned by Rufus in the GitHub thread <#154>), incl: https://pathwaystoliberation.com/the-matrix-instructions/ - And following up on Rufus' email suggesting a full dump of IDGs into a structured form e.g. questions, categories etc. - *Data Research*: What design elements support the data research opportunities in transformative experiences. For example, how do the assessments separate and compare the contrasting dimensions of personal work? Throwing a lot out there, but would be delighted for the above to be a domain of critique / exploration, and hear thoughts from the group on such approaches and design criteria. Happy to follow-up on any questions / threads. All the best, John

…

On Mon, 6 Feb 2023 at 12:01, Liu3ow ***@***.***> wrote: For assessment of programs the big need is indeed to collect some materials for assessment, as discussed during the talk by @baouroux <https://github.com/baouroux> . John, if you have some links on best practices to share on - how to collect the "warm data" or "unstructured data" from narratives - how to make surveys for such programs Thank you! — Reply to this email directly, view it on GitHub <#154 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A23MU4B2LKBI6NRWFYTZ4ADWWDKXXANCNFSM56FHMZNQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Liyubov]

Thanks John, really helping to start some reading.

I am also trying to be careful with using the vaguely defined term "warm data" originally we heard about it from https://warmdatalab.net/
In this context this term was related to:

• sharing something which is hard to quantify (in linear or whatever scale)
• sharing something which does not necessarily come from the survey

[baouroux]

Hi Liubov,

Thanks for the chance to explore these topics on our group call today...

Very topical that you mention in your latest message Nora Bateson's Warm Data Lab, since my LinkedIn post I spoke about on the call, was in response to Nora Bateson's 'viral' Facebook thread on the controversies surrounding developmental psychology models.

Whilst there are some fundamental questions to ask about the way that frameworks such as Spiral Dynamics are being used, Nora's generative provocation in categorically arguing against all developmental frameworks increased the urgency to write this LinkedIn post referring to my draft paper to offer the concept of 'developmental sensemaking'.

Look forward to discussing further, and tackling these fundamental questions of measurement!

All the best,
John

[baouroux]

Hi again Liubov, I forgot to actually pick up on your thread leading to the question, what is "Warm Data" as Nora defines it. I heard Rufus imply on one of our earlier calls the question, is Warm Data just contextual data? I've tuned into Nora's interviews over the years (not yet read her book), and know a couple of people who have participated in her Warm Data Labs. Below is perhaps a useful definition from her interview with Jim Rutt: https://www.jimruttshow.com/nora-bateson/ - 54min into the podcast: "the definition [of warm data] is that it’s transcontextual information about the relationships that are part of a complex system." - She goes on: "....I developed this thing called a Warm Data Lab. And in the Warm Data Lab, you have a group of people and they explore a question that involves a complex system. So you could ask the question, for example, like what is health in a changing world? And there’s lots of contexts that are involved in that question. It could be an economic context. There’s certainly technology, culture, family, politics, history, technology, art even, there’s sexuality, there’s all sorts of contexts. So the way it works is you have a group of people in a room and they are exploring this question, and all those contexts are laid out like at different groupings of chairs. And then people just kind of move whenever they want to between the chairs. So they might start off talking about health in a changing world through the context of education. And then whenever they want to, they move to another place and they look at the health. And what happens is after they’ve changed contexts a few times, they start to have the experience of the interdependency that you don’t actually change health by adjusting any of the particular contexts and how one then goes about contributing to the health of one’s community or nation or city or whatever becomes a very different question." These rotations between discussion breakout groups, that look at complex issues from multiple perspectives in quite spontaneous conversational ways, is I understand the DNA of her Lab sessions. She then goes on to describe the crucial next stage of the whole group coming together to piece all the threads together, and 'make sense' of all the discussions. She has a fancy word I think for this co-learning phase, as 'Schismogenisis'. Just to make a link to my paper that speaks to the controversies in developmental psychology, I'm proposing/researching that there may be measurable cognitive constructs from developmental psychology (3rd person) that arise in this last phase that Nora mentions, that follows from the initial breakout group phases (2nd person). The cognitive constructs therefore could be seen as supporting the grammar of our sensemaking - a reference that I'm taking from John Vervaeke, and stays faithful to a Dave Snowden principle of "practice led theory" - i.e. the 2nd person leading the 3rd person. Hope you don't mind my throwing this out, as part of my own process of testing my understanding and ideas! All the best, John John Oliver Mobile: +33 631 60 2994 Bordeaux, France

…

On Fri, 17 Feb 2023 at 14:43, Liu3ow ***@***.***> wrote: Thanks John, really helping to start some reading. I am also trying to be careful with using the vaguely defined term "warm data" originally we heard about it from https://warmdatalab.net/ In this context this term was related to: - sharing something which is hard to quantify (in linear or whatever scale) - sharing something which does not necessarily come from the survey — Reply to this email directly, view it on GitHub <#154 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A23MU4FWSXEJIFVOPFQWPSDWX556XANCNFSM56FHMZNQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[1000Folds]

Hello @baouroux @Liyubov and all, Really interesting and stimulating line of thought here and I've been sitting with some of the questions raised on the call and am trying to find a non-influencing way of measurement. Intuitively, I feel it may not be possible due to entanglement and ontology.

Some reflections:

• John, am really into the 1st and 3rd person inquiry. Got me thinking of my search for the 'we' archetype. What arose is the 1st person already has a we to go along with the I. Might the 3rd person 'it'ing' of the 1st person we; be the ground for the 'we' archetype to arise? (this takes some work as you have to hold both the 1st and 3rd person simultaneously and non-dualistically).
• Warm Data (Nora is an old friend) is not contextual, it is transtextual. It relies on the relationships between and among, often at the edges of systems. The sense making is accomplished through abduction, learning/understanding one system through another - this is the heart and process of warm data labs. 'Schismogenisis', is from Gregory and is the process of creating division, not ideal in society. It is the recursive loop of A acting in response to B acting in response to A . . . . essentially, a mechanism for entrenchment and polarisation.

Looking forward to continuing this rich discussion!

Joe

[1000Folds]

Typo - read: 'trans-contextual' for 'transtextual'.

[baouroux]

Thanks for the further reflections Joe, and for the clarifications on Nora's terms. How interesting that you know her, and look forward to bringing your threads together in the following calls. On your first point, I'm currently reading an author on the creative experience as a spiritual practice, and I've found really interesting her emphasis of the 'hermeneutic arcs' in meaning making, i.e. the movements between perspectives (1st, 2nd, 3rd person)... Which also reminds me of a Dave Snowden complexity principle of looking at the vectors vs the coordinates. Cheers for now, John

…

On Sun, Feb 19, 2023, 12:30 1000Folds ***@***.***> wrote: Hello @baouroux <https://github.com/baouroux> @Liyubov <https://github.com/Liyubov> and all, Really interesting and stimulating line of thought here and I've been sitting with some of the questions raised on the call and am trying to find a non-influencing way of measurement. Intuitively, I feel it may not be possible due to entanglement and ontology. Some reflections: - John, am really into the 1st and 3rd person inquiry. Got me thinking of my search for the 'we' archetype. What arose is the 1st person already has a we to go along with the I. Might the 3rd person 'it'ing' of the 1st person we; be the ground for the 'we' archetype to arise? (this takes some work as you have to hold both the 1st and 3rd person simultaneously and non-dualistically). - Warm Data (Nora is an old friend) is not contextual, it is transtextual. It relies on the relationships between and among, often at the edges of systems. The sense making is accomplished through abduction, learning/understanding one system through another - this is the heart and process of warm data labs. 'Schismogenisis', is from Gregory and is the process of creating division, not ideal in society. It is the recursive loop of A acting in response to B acting in response to A . . . . essentially, a mechanism for entrenchment and polarisation. Looking forward to continuing this rich discussion! Joe — Reply to this email directly, view it on GitHub <#154 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A23MU4FKK4AF73MQZGATP4LWYH74DANCNFSM56FHMZNQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[baouroux]

Hi Joe, Thanks again for picking up on the schismogenesis definition... https://en.wikipedia.org/wiki/Schismogenesis. I find fascinating the notion of these feedback loops that (exponentially?) amplify differences. "Symmetrical" schismogenesis as Gregory Bateson defines it, kind of reminds me of the escalation of disputes between my young kids!!! This might be quite a stretch, but I wonder if you could even draw a parallel to the notion of "reciprocal narrowing" that John Vervaeke quotes from Marc Lewis (as applied to the phenomenon of addiction), within our own physical/physiological self-systems. And the inverse - to the topic of measurement of transformation, and even in the We Space - that could be seen as *reciprocal opening* dynamics. Could that have a name, for us to define the polarity of schismogenesis vs. (for example) coherence-genesis / resonance-genesis / mutuality-genesis? Measuring such feedback loops could be an interesting lens on the transformation process, vs. measuring the more complex/divergent domain of transformation outcomes? Cheers, John John Oliver Mobile: +33 631 60 2994 Bordeaux, France www.tyler.world (art practice) www.interiortruth.com (witnessing practice) www.collectivenavigator.com (dialogue platform) *LinkedIn* <https://www.linkedin.com/in/john-oliver-bb01a/>

…

On Sun, 19 Feb 2023 at 12:30, 1000Folds ***@***.***> wrote: Hello @baouroux <https://github.com/baouroux> @Liyubov <https://github.com/Liyubov> and all, Really interesting and stimulating line of thought here and I've been sitting with some of the questions raised on the call and am trying to find a non-influencing way of measurement. Intuitively, I feel it may not be possible due to entanglement and ontology. Some reflections: - John, am really into the 1st and 3rd person inquiry. Got me thinking of my search for the 'we' archetype. What arose is the 1st person already has a we to go along with the I. Might the 3rd person 'it'ing' of the 1st person we; be the ground for the 'we' archetype to arise? (this takes some work as you have to hold both the 1st and 3rd person simultaneously and non-dualistically). - Warm Data (Nora is an old friend) is not contextual, it is transtextual. It relies on the relationships between and among, often at the edges of systems. The sense making is accomplished through abduction, learning/understanding one system through another - this is the heart and process of warm data labs. 'Schismogenisis', is from Gregory and is the process of creating division, not ideal in society. It is the recursive loop of A acting in response to B acting in response to A . . . . essentially, a mechanism for entrenchment and polarisation. Looking forward to continuing this rich discussion! Joe — Reply to this email directly, view it on GitHub <#154 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A23MU4FKK4AF73MQZGATP4LWYH74DANCNFSM56FHMZNQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[1000Folds]

Hi John,

Yes, I think that John V's reciprocal narrowing is a 'schismogenic' process of some type. And good that you point out the two types, symmetrical and complementary, the one in my example. Though I'd be willing to wager your kids engage in both types ;).

I like where you are going with this inquiry of measurement in the We space; I wonder which We space you mean, 1st person or what I suggested, 1st person from/with 3rd person perspective? Either way it strikes me that a reciprocal opening dynamic can exist.
Do you see it differently, and if so - how, from a societal fade, or trend? Is that the same as coherence-genesis / resonance-genesis / mutuality-genesis? My gut says no, and that you are gesturing at something other or larger or more inclusive. I'd be delighted if you can say more as I think this is very rich.

And this is for everyone, at the semantic level - do we all mean or have the same understanding of 'measurement' vs. 'assessment?

I see measurement occupying the past. We measure what is there, what already happened. What would be a measurement in the future?

Assessment has more of a participatory flavour to me, adjacent to what had happened, is occurring, but how reliably indicating what will emerge? Again, like what I said on the call, somehow measurement and maybe to lesser degree, assessment lands us in a reductive/dualistic space. That could be useful for complicated systems, but I understand we are addressing in this space, complex systems.

Your process vs outcomes, John is interesting - I'll have to sit with that, in this context, a bit more :).

Cheers,
Joe

[Liyubov]

Following on the presentation by @asimong on 7th July there are some things to be discussed and links which were shared in the conversation:
by simon https://docs.google.com/presentation/d/150FGIuh9kBxP3wJI7xiRpxSZAy3ejxSGvDyMSzdoL7M/edit?usp=sharing
and his talk on assessment of https://www.innerdevelopmentgoals.org/framework
by boaz https://carl2030.org/ The Competency Assessment for Responsible Leadership

looking forward for discussions and concrete research questions on it!

[asimong]

Thanks Liyubov! My sense is of a need to meet together and through conversation to identify some common ground where we share interest. The area as a whole is so large, I would not have thought we can take it all on!

[asimong]

Following on the presentation by @asimong on 7th July there are some things to be discussed and links which were shared in the conversation: by simon https://docs.google.com/presentation/d/150FGIuh9kBxP3wJI7xiRpxSZAy3ejxSGvDyMSzdoL7M/edit?usp=sharing and his talk on assessment of https://www.innerdevelopmentgoals.org/framework by boaz https://carl2030.org/ The Competency Assessment for Responsible Leadership

looking forward for discussions and concrete research questions on it!

I think I have to contribute here in order to be able to be found and referred to! I'm (as usual) asimong

[rufuspollock]

@asimong fixed.

[catherinet1]

@karlsteyaert , imagine you'd be interested in this discussion thread