Re: Adding Org Files to org-agenda-files

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-12-13 18:33]:
> Dear Jean Louis,
> 
> Thank you for the detailed insight into your extensive experience of
> project management and practical planning. I do not have that much
> experience, but can provide a significantly different point of view
> related to my research work.

I like to know it all to collect or steal what is good. ;-)

I am researching other software too and collecting what is good.

> My personal experience is doing a lot of research work. That's probably
> on the other side of the spectrum from the environment you are working
> in. I cannot define very concrete steps to execute a research project.
> Not because it is impossible, but rather because failures are pretty
> much guaranteed far before all the steps are executed. Moreover, most of
> time, it is not possible to consult someone else on resolution of the
> problem causing blockage, simply because the problem is something that
> never ever appeared in the past (that's the whole point of doing
> research). Instead, I need to spend a significant time trying to find
> *similar* problems digging through literature, talking to people working
> on related problems, or even just thinking. Then, waiting until the
> solution appears becomes a waste of time (there is even no guarantee
> that solution exists) - if there are other alternative approaches to
> achieve the global project objective, they would better be tried before
> the blockage in one particular direction in solved. In fact, switching
> to alternative approaches (or even projects) sometimes help to look at
> the problem from different angle and solve it. The described difficulty
> is *underestimation* of what can happen - even the initial project
> objectives can be changed according to the current research results.
> Trying to stick to a strict project structure in such a situation is a
> waste of time - project must be re-created from scratch very too often,
> unless it is more flexible from the very beginning.

When researching is not conducting clear plan. It is different stage
or can be independent activity of some planning with firm
objectives. I do research too. Plans are based on research
information. I do not mix research information and planning based on
research and that way plans remains without disturbance. They can be
read on screen what people do, executed from mobile phone or from
printed papers.

More interesting hyperlinks:

Planning For People Who Suck At Planning
https://debbieinshape.com/suck-planning/

How to STOP Over-Planning (And Start Doing!)
https://eringobler.com/overplanning/

Planning fallacy: why people suck at planning
https://blog.sandglaz.com/people-suck-at-planning/

The last above also speaks of our human over or under estimations. To
make things go right one has to have good sense of reality. Otherwise
schedules and deadline never become what they are intended to become.

> The described situation is where NEXT tasks/projects can become
> extremely helpful. Multiple NEXT tasks do not mean that I need to look
> at them every day and switch from one to another. There are NEXT tasks
> and there are NEXT tasks that are actually scheduled on specific day.
> One day cannot have more than several (ideally one) NEXT task (possibly
> containing a checklist). That's where agenda comes handy. It is not used
> to decide what to do during that day. It merely shows earlier decision
> when planning which project (and corresponding doable NEXT task) to do
> on specific day. Other items in agenda are things that must be done on
> that day anyway (meetings, mandatory habits, etc). Polluting agenda with
> unnecessary staff is no better than mindless browsing of youtube.

I do agree that thinking may be helped, just that as "next" I can have
only one next by its meaning of immediately following by its time or
order. I do not mind you having that type of thinking as you relate
your meaning to the text, tags or anything you have in front of
you. My point is that NEXT will not mean to onlooker the same what it
means to you. It would require explanations and learning the habit as
I always look on it how a group of people would look on it.

The nodes in my meta level dynamic knowledge repository have "Set ➜"
and by using that arrow I know I can enter into the node, it is not
just a note, I can go into the subtree. Such marking with the arrow
would be more logical but NEXT to me.

What is next probably means it is doable something and there may be
multiple doable items.

The items we put in planning are all doable. We do not put those not
doable as such are chunked into steps until each step becomes
doable. There is no block or obstacle for staff on ground in reading,
passing over, executing those steps as each step is doable for 
them. They know what is next by simple looking what was done
previously. The true meaning of "next" is used there.

Some small purchasing projects may give direction naturally which shop
is next to be visited for water pump to be purchased.

I have many geographical locations and when I am in huge country like
Tanzania I may like to visit some of locations that are closer to my
location. That require program to run over all locations to calculate
what is closer geographically.

Sure that there are different paradigms. I will probably never mark
something as NEXT, I will just take that what I think I can do and do
it. No marking. If I mark it, it was probably not next. That means
marking A next, B next, C next... but if it was really next why I did
not do it... this comment comes only from the true meaning of "next"
as one word. As term of multiple words "NEXT AVAILABLE TO DO" it would
give better meaning.

> I hope I described my use-case sufficiently to show the difference with
> your situation. For research, "fully understanding all parts of the full
> project" means that project is pretty much completed and there is no
> need to look further except maybe writing reports.

Definitely I understand that and I agree. It is difference in marking
and helping people understand the meanings.

To fully understand all parts of project document is better
formulated. All our projects are also research projects. We come into
area where we do not know circumstances and have to collect
information about people, technicians, medical facilities, people's
needs in the village, safety, crime, dangerous animals and their
probability of appearing and behavior, distances to roads, water
sources and their volume, vegetation, land configuration and so
on. For all those questions we have a written document. When there is
any word that person does not understand in the project it has first
to be clarified. People think they understand but they don't. We make
sure they will not go to field without truly understanding what has to
be done. Not the results or reports of the project but what is written
in the document.

> As I mentioned earlier, the purpose of NEXT items is not for daily use.
> That's where scheduling can be used (at least, in my workflow). The
> purpose of NEXT items is making project review easier - they are mainly
> needed to provide hints on decision how to proceed with a blocked
> project. As you mentioned, this is useless when project steps are
> well-defined and little trouble is expected during execution.

Projects may become blocked. As I think in this moment they become
blocked especially when project items are not fullfiled on
ground. When people become shy, timid for some reason or think they
should not do what is written or ordered or advised. That is where it
gets blocked on my side. It happens. Then new people have to be
chosen. 

> If target is flexible (like in research), extra TODO items can be useful
> as a reminder what else might be done. Also, note that org-mode does not
> strictly force todo dependencies. One can always force unconditional
> todo state change with C-u C-u C-u C-c C-t (or by setting
> org-enforce-todo-dependencies and
> org-enforce-todo-checkbox-dependencies).

Extra items fit into personal information management. As soon as there
is collaboration one has to think well what one writes as not to cause
further misunderstoods and not necessary discussions.

Our project documents have usually enough space on the paper so people
write those notes, observations on the paper itself. Later it is
scanned. Important information is reported in a daily report and from
there refiled maybe. Some new people are entered into database, new
locations or structured reports. With the collaborative online
accessible database, people may write reports offline and then click
something to upload it to the database.

In that sense we collect knowledge, reminders, etc. Just that main
project as document remains clean. If anything need enhancement in
such document new step is then added easily. I have always kept
documents separate and clean from each other and that is why I have
not get scattered things that I need to search through.

> >   I am using word "reasonably" as that involves human who decides
> >   about it and not robotic following of the tasks and executing them
> >   just because they may appear as not DONE.
> 
> I look at it from different perspective. Task dependency is forcing me
> to double-check the tasks not marked done and explicitly thinking if I
> need to do them and improve the project (remember, there is no
> well-defined project goal for me - things can always be improved, unless
> there is time limit).

On my side I made it very easy to make any hyperdocument
ACTION-able. But I made it harder to complete it or to remove action
as then Emacs ask me: Do you really wish to complete?

There are those 2 different paradigms to first close tasks and then
close the senior heading or purpose, or to mark all tasks redundant
when purpose have been accomplished.

As I know work with the database it is easy to assign such type when
purpose have been accomplished that all subordinate items become
redundant or unmarked from being any actionable items. I do not mark
them anyhow as if they are under project which is considered action
there is no need to mark each task which is visibly task to be task by
marking it with TODO. If necessary I can even highlight whole lines,
but no need. I could include sound or even spoken text that there is
something to do in that section of the tree. If headings are well
written computer may also wake me up by alarm and talk to me what is
to be done next.

Now back to those 2 paradigms, don't you think that sometimes it is
necessary for heading to actually complete all subordinate items to
complete the senior item? And then sometimes it is necessary to mark
all subordinates redundant when senior item is marked as done?

And then we have variables that impact not only one Org file, but all
Org files together. That does not fly well. Each heading could be of
this or the other type and should have its property marked as this or
that. Maybe this exists in Org mode that I do not know.

** Paint the room

1. Purchase paint
2. Purchase brush
3. Find newspaper
4. Paint the room
5. Clean newspapers
6. Clean everything

That above is example where subordinate items need to be accomplished
for senior heading to be marked as done.

** Paint the house

1. Ask Joe if he could paint the house
2. Ask Jane if she could paing the house
3. Ask Jimmy if he could paint the house

The above example is where subordinate items need not be completed for
the objective to be fullfiled. One Org file could have separate types
of headings where one general option does not fit all cases. 

> If I decide to not do the not-done task (by actively thinking, not
> by mindlessly marking project done just because I think the goals
> are nominally achieved), I just mark the task CANCELLED (which is a
> type of "done" keywords in org terminology). At the end, task
> dependency allows to double-check for any missing ideas I could
> forget about.

Yes, it is on that part of marks. Some things get cancelled. I am
using various keywords to mark the items or assign some meta data, but
there is main status of task that says if action is to be done. If
something is cancelled, action mark is removed and timestamp for
cancelled inserted. If it was cancelled and re-activated there is
track of it as well to see who did what at what time and why
(description possible). As we can see from past few weeks meta data
for a heading becomes very large and it is not made for plain text, it
needs separate database, so I have it.

> > What is not reason is to have unreasonable files of allegedly ordered
> > tasks which are in reality not ordered and proof for that is that
> > org-agenda exists in the first place. People do not keep their
> > projects and tasks in ordered manner and they need org-agenda.
> >
> > That is why I almost never used org-agenda in last 5 years.
> 
> While reading your examples about why org-mode is often promoting
> procrastination and messed up organisation, I feel that you expect more
> from org-mode than it is.

I have expected it to enhance something in my planning since 5
years. It did enhance the document preparation, but also in same time
degraded document preparation. Sure that I am not limited in how much
LaTeX I can use but if I wish to have readable document I have to keep
things simplers with Org than with LaTeX. That is personal benefit for
me. We make contracts with Org mode, project documents, instruction
documents.

Sure I expected it would bring some better paradigm of managing
tasks. As you have seen in the manual it teaches users to put notes
and tasks anywhere, somewhere it mentiones best seller book and GTD,
nothing more than that. Other websites speak of various paradigms. In
Org they are not explained. Org mode is pretty much low level, not
high level. It is a mark up language limited to Emacs.

I do not see that I expect too much of Org. The CRM software I was
researching 15 years ago and they still have people assigned, related
people, sending tasks by email and so on. Those are features that
should be commong to any task or planning management. It is not Org
mode only, there are other software designed without enough
brainstorming.

Summary is that Org mode is good as PIM = Personal Information
Manager, not for anything beyond that.

As I love Emacs I got hooked on how people explain their Org mode and
tasks and stuff and I was thinking that is something. People manage
mostly their personal life, get hooked on it due to love to Emacs. And
I am focusing on groups and people beyond my individual personal
information management. If it does not work well alone, then I have
upgraded it that it works from meta level. Org remains there as one of
possible mark up types or modes that one can use.

Database is Emacs independent. SQL queries are pretty much same and
functions may be reused by other software. Let us say I wish to enable
Javascript access, or make Android/LineageOS/Replicant application, or
even use it from vim or some other applications or different OS-es it
is possible. Haiku is quite nice OS, there is Emacs if I remember
well, it could use the database. Mezzano is Lisp OS:
https://github.com/froggey/Mezzano and it would be possible to
re-write Emacs Lisp functions to access it from such.

People can even access the database from Org mode itself, and update
the nodes as well. I could write simple functions to convert Org mode
notes to database nodes. Then it could as well edit database nodes. It
is even simpler:

- export from database, each heading would have its corresponding and
  really unique ID numbers. As long as not changed by hand, simple key
  press like F5 could update the edited node back into the
  database. Additionally it would not need to, it could remap file
  saving action or some kill-buffer or similar to iterate over all
  nodes, compare them and update some into the database. Org mode
  could be used to edit database entries.

> You provided examples that people used their brains instead of
> computers and paper instead of files in the past and successfully
> managed complex projects. I would like to point out that org-mode to
> organisation and project management is just like pen and paper to
> project management and organisation. It is easy to have paper notes
> scattered all around the office, home, and half of them lost
> somewhere.

I never had it that way. I have been keeping those hard binded
planners with dates and years, but I preferred those without dates for
liberty of using multiple pages and I wrote things there. So I never
had in my life situation of scattered papers. One year would have 1 or
2 such books. Later I was using German style notes on paper that are
all ordered in one box. This related to people and closing agreements
to people, everything was on one place. Some phone numbers from phone
calls were written onto some notes and rewritten later. More than that
I never had scattered paper notes around office or anywhere. I do have
scattered papers like documents that need sorting, but are more of
archive type.

That is why I say that Org manual section on TODOs is teaching people
bad habits and admits it is bad habit as it speaks of scattered stuff
and compensation for that by using Org agenda.

> Same in org-mode, and you provided enough examples. One needs to
> have a proper mindset and established workflows to manage real
> projects with pen and papers.

That is what I expect from the Org manual to give people more options
to establish their workflows as now it is giving options too but
narrowed and one can see that people follow the pattern and paradigm
that leads to more and more complexities. Life does not get simpler
that way, it becomes complex and dependent on computer very much.

Would those scattered files be printed one would not know where is
what, as there is no real life agenda.

My files when printed are ordered by projects and one can pick out a
project and follow it to fullfill its purposes and goals. There is
nothing missing in the project neither there are supefluous items that
belong to some other project. Agenda is right there by looking into
the project itself.

Maybe that should be the test if planning is good, if one can print it
and not get lost without using computer.

> I think about org-mode as about improved pen and paper - with proper
> workflows and organisation it can be very efficient;

That is right. But that is what people do not get to learn.

> without organisation - it's just a digital mess, worse than some
> computer desktops.

I don't have Desktop polluted, it was always empty. I do have one
folder in $HOME like TO-DO and one TO-SORT where items go before
sorting.

> org-mode provides a set of instruments - they can be used in vastly
> different project management styles, some are more suitable to
> specific styles, some are less suitable. As you mentioned,
> org-agenda is not suitable for your style. It can be much better for
> others.

If there are paradigms that can help people such shall be referenced
from the manual and pages. There is already reference to GTD, I do not
find myself there but if it helps some people those paradigms should
be explained. Otherwise without explanation people are learning what
they do, I can see many do not have organized way like you.

> While agenda can certainly show such kind of mix, it is indeed very
> inefficient use of this tool. If other readers of this thread are
> interested in better practices on using agenda, I recommend what is
> recommended in [1]. It is absolutely crucial to keep daily agenda as
> small as possible - only tasks that must be done on that day *and in the
> location context* should be shown. Mixture of home and work tasks must
> not happen. I knew this when I just started playing around with GTD, and
> I thought that it is not important. After years of experience, I have to
> say, that the rules about agenda are determinal to finishing work that
> matters.

Also nice insight.

> [1] Allen David [2015] Getting things done : the art of stress-free
> productivity

Natural Project Planning with org-mode
http://members.optusnet.com.au/~charles57/GTD/Natural_Project_Planning.html

On that link it speaks of GTD principles, one is defining
objective. But among those steps I do not find myself there except
number (1).

1. Outline objective
2. Outcome visioning
3. Brainstorming
4. Organizing
5. Identifying next action

This because visioning and brainstorming would come before the
objective is well defined. Some rather general or vague objective is
there during visioning and brainstorming. When well defined objective
is written down, those parts of organizing and identifying next
actions become part of the project. If money is missing, project step
will say that money has to be made. If people are not recruited or
employes, project step will dictate that. I was never lost after (5)
like identifying next action that I can correlate to the above
system. I have defined ALL actions necessary to achieve the goal by
deduction from the objective and never had one or more actions by not
knowing what would be next. Then it is not planning, it is
adventure. I do that in my mind and discover life that way.

Planning is when set of factors is already known, not enough known,
but pretty well known and if they have to be discvered that discovery
becomes part of project that can itself demand project creation at
certain stage or branch. 

> > Working on Org file means working from bottom to top:
> >
> > - make tasks, little here, little there, organize maybe by some
> >   groups, make this or that file, search through agenda because I have
> >   not ordered anything how it should be. Think of task first because
> >   it is scheduled for its own sake of being scheduled. Do the task
> >   because it is task and not part of one higher purpose. Mark flag,
> >   add properties, tag them to be able to search them.
> >
> > The Org way of doing things is organizing procrastination with more
> > and more increasing complexities that are allegedly supposed to make
> > life easier.
> >
> > Please do not stone me.
> 
> While one can work with org file the way you described, it is not
> necessary (and should not be done most of the time). High-level planning
> is very important. It can be ignored to capture ideas in the middle of
> doing something else, but those captured ideas should be thought about
> in context of the whole project and placed into (or discarded from) the
> project according to top-level objectives.

I agree completely.

> > Here is structure of a project, as part of bigger plan. Projects can
> > be structured any how on my side. When assigned to other people there
> > are sections of introduction:
> >
> > 1 Primary principle for reading ;; explains to people not to skip 
> > misunderstoods
> > 2 Primary principle for communication ;; that we shall collaborate, etc.
> > 3 Definitions of words ;; defines terms related to project
> > 4 About company
> > 5 Goal of the project ;; known objective, actions are done to achieve
> >                          the goal and it has clear quote
> > 6 Purpose of the project ;; A purpose is a lesser goal applying to
> >                             specific activities or the sujects. It
> >                         often expresses future intentions
> > 7 Requirements for this project ;; no moving to "TODO" without it!
> > 8 How to do this project ;; explains how to conduct project, reason,
> >                             logic, collaboration is all here
> > 9 How to report        
> > 10 How to report on events
> > 11 How to make pictures
> > 12 Communication requirements [0/16] 
> > 13 Personal introduction
> > 14 Project steps            ;; this is where operational targets are defined
> > 15 Awards
> 
> Note: This project template is fairly similar to what is recommended by
> Allen David, except reporting and communication. I lack experience of
> large collaborations, so cannot elaborate much on this part.

Just think how would you get feedback of somebody on distance? If they
do not report before action, during action, after the action, you may
get lost. If there is no daily report, you will not know that person
got stuck somewhere across the river in a truck because it was raining
and you would demand that same person to be in a city 100 miles away
negotiating with your partners. In other words when there is no
coordination things do not get done properly.

> > If things are well organized from ground up then agenda becomes
> > redundant.
> >
> > Organized implies to me to know what is next to be done.
> >
> > Unorganized person does not know what is next to be done. That is why
> > Org agenda is there. Because tasks are scattered, not organized.
> 
> Agenda cannot help unorganised person.

Come on, let's face it, agenda is exactly for that. OK joke on side. I
do think that organized person have agenda, it is list of things to
do, but that list is right there and is not a product of search
through scattered files and notes. I was writing back in time my
weekly agenda and pushed all items to be done. Something was not done
and was pushed to next week to be done. This worked well. Nobody
doubted it. It was one paper, and each staff member had the
paper. Supervisors could simple ask for the weekly agenda and would
inspect it or advise enhancements. Imagine if we would need to invoke
computer search to find our weekly agenda... funny situation. 

> Similarly with a paper (or paper calendar) that cannot help
> unorganised person.

True.

> However, either calendar or agenda can be used efficiently as tools
> helping organisation (when they are suitable for the specific
> situation).

Agenda can be used surely to help organization of personal things,
maybe organization of people, rather unlikely as it is more for
personal information stuff.

> > Org mode has headings and hierarchy and established ways for people to
> > order their goals, projects, tasks, but it is not what people are
> > doing, because there is no form structure in Org mode to tell where
> > something is allowed to be ordered and where not.
> 
> Well-organised person would not need computer to keep records in
> relational database - even a simple paper would do if used properly
> [2].

Definitely. But because my hand writing would be more readable to crow
than to human, I have to keep things in computer that has nicer output
on printer. In case of nuclear war or aliens superpower and computer
outages I will have to switch to Zettelkasten again. Before I used few
boxes with Zettelkasten and it worked well. But I did not have too
many relations.

When new inquiry arrives over website, it does not arrive to database
straight, but as Lisp data. I press key in mutt that lisp data gets
decrypted and inserted into the database. There is phone, full name
parsed, country, email address and inquiry text, there are many
inquiries, I would not catch up writing this all. One key on computer
writes their names, I need not.

> org-mode provides such tools, but org-mode does not teach or enforce
> organisation.

Wrong. (info "(org) TODO Items") teaches disorganization.

If it already teaches some disorganization that is compensated by Org
tools, then why not teach better some methods of organization. Why
leave it to other people when it could be centrally done from a
website at least. Paradigms of organizing life.

> The cost of being flexible is possibility to misuse. The power of
> being flexible is possibility to use much more efficiently than more
> restricted tools.

Which brings it to the level for advanced users only. It does not
integrate things for average users. The whole point of computer is to
integrate things for human and have human work less, not human work
more for software it uses. 

> > Thinking on long-range goal helps in determining short-range goals,
> > which help in determining which projects or tasks are to be executed.
> 
> One can also refer to GTD methodology, which is more about long-term
> goals than about individual task - the point many people miss. (Search
> for GTD: Purpose, vision, goals, and areas of responsibility + weekly
> review).

It is something idiosyncratic, I do not relate to it, only the
objective is to me known. Those other things are upside down to me. It
shows how people are different. I would not give that book to somebody
as I could not recommend it that way. For various business and
humanitarian projects brainstorming was before project planning, not
after. Before the goal definition, not after. Visions as well. Then
came goal definition and all the plan can be written from mind without
even delaying there. Personal habit brought me in situation that I
could take almost any business that I can see in front of me, review
its statistics, gather information, and break down the projects
without delay on how to improve the business.

If I am in situation not to have project in front of me, that means I
was not planning. If I need brainstorming at the time when I need
action, it may be too late. Action shall be swift, clear, unmistaken
with clear objective to be achieved.

I do not condone wars, but imagine if military commander gets into the
situation to know that one needs to enter some area, but let him
brainstorm there and create visions, maybe he finds out what is next
to be done. I know many did so, and died. That is why successful
military operations have possibly all circumstances planned including
those on what to do when it is not planned. 

Regarding SQL it is pretty readable as human language, probably few
times better readable than well written meaningful Emacs lisp
functions. SQL databases spare a lot of time to programmer as they are
already programmed with many functions.

SELECT * FROM notes ORDER BY notes_datecreated DESC;

would sort notes by date created (call the column as you wish) but in
reverse order. Latest notes would be shown first. Order by its ID, or
by scheduled, deadline, order by people assigned, you name it. It
helps to get those intersections what agenda is trying to do with
hardcoded functions. In general it is easier for people to extend a
report by constructing SQL then by constructing Emacs Lisp functions
due to its meaningful language:

UPDATE notes SET notes_text = 'My new update' WHERE notes_id = 2;

I am sure you can understand that above without previous SQL
knowledge. it updates text of the note number 2 to something
new. Those SQL queries are low level and usually not visible to
users. Users chooses Emacs function from menu or M-x or by any other
means from other software. SQL is executed as result of Emacs Lisp
function.

Thank you for conversation,
Jean