WIP: Draft about the profiler

doc/devel/rfc_pending/ResourceManagementProfiling.txt

@@ -0,0 +1,167 @@
+Resource Management: Profiling
+==============================
+
+// please don't remove the //word: comments
+
+[grid="all"]
+`------------`-----------------------
+*State*         _Idea_
+*Date*          _Fri Jul 23 19:34:29 2010_
+*Proposed by*   Christian Thaeter <ct@pipapo.org>
+-------------------------------------
+
+[abstract]
+******************************************************************************
+From the beginning on we planned some kind of 'profiling' to adapt dynamically
+to workload and machine capabilities. I describe here how statistic data
+can be gathered in a generic way. This will later work together with other
+components tuning the system automatically.
+******************************************************************************
+
+
+Description
+-----------
+//description: add a detailed description:
+
+I just introduce some ideas about the planned profiling framework here, nothing
+is defined/matured yet this is certainly subject for futher discussion and
+refinement.
+
+.Requirements/Evaluation
+ generic::
+	Profiling should be sufficiently abstracted to have a single set of
+	datastructures and algorithms to work on a broad range of subjects being
+	profiled. Moreover the profiling core just offers unitless counters,
+	semantic will be added on top of that on a higher level.
+
+ least possible overhead::
+	Profiling itself must not cost much, it must not block and should avoid
+	expensive operations. Simple integer arithmetic without divisions
+	is suggested.
+	
+ accurate::
+	We may sample data on in stochastic way to reduce the overhead,
+	nevertheless data which gets sampled must be accurately stored and
+	processed without rounding losses and drifts.
+
+ transient values::
+	It's quite common that some values can be far off either in maximum or in
+	minimum direction, the system should adapt to this and recover from such
+	false alarms. Workload also changes over time we need to find some way to
+	measure the current/recent workload an grand total over the whole
+	application runtime is rather uninteresting. While it is also important
+	that we adapt slow enough not to get into some osccilating cycle.
+
+ active or passive system::
+	Profiling can be only passive collecting data and let it be analyzed
+ 	by some other component or active triggering some action when some limits
+	are reached. I am yet a bit undecided and keep it open for both.
+
+
+	
+
+
+
+
+.Brainstorming in Code
+[source,C]
+------------------------------------------------------------------------------
+
+typedef int64_t profile_value;
+
+struct profile
+{
+  ProfileVTable vtable;
+
+  /*
+   Using trylock for sampling makes it never contend on the lock
+   but some samples are lost. Should be ok.
+  */
+  mutex_t lock;                         /* with trylock? */
+
+
+  /* statistics / running averages */
+
+  /* n being a small number 2-5 or so */
+  profile_value max[n];                 /* n maximum values seen so far, decreased by recovery */
+  profile_value min[n];                 /* n minimum values seen so far, increased by recovery */
+
+  /* store sum & count, but average calculation implies a division and will be only done on demand */
+  profile_value count;                  /* count profile calls */
+  profile_value sum;                    /* sum up all calls, average = sum/count */
+
+  /* current is the sampled value to be integrated */
+
+  /* trend is caclulated before theb new run_average */
+  profile_value trend;                  /* trend = (trend + (run_average-current))>>1 */
+
+  /* we may need some slower diverging formula for running average */
+  profile_value run_average;            /* run_average = (run_average + current)>>1) */
+
+
+  /* active limits, define whats good and whats bad, calls back to vtable when limit is hit */
+  profile_value max_limit;
+  profile_value min_limit;
+  /* do we want limits for trends too? */
+
+  /* we count how often we hit limits, a hit/miss ratio will give a good value for optimization */
+  profile_value hit_cnt;
+  profile_value high_miss_cnt;
+  profile_value low_miss_cnt;
+
+  /* recovery state */
+  int rec_init;
+  int rec_current;
+  int rec_percent;
+
+
+  void* extra;
+};
+------------------------------------------------------------------------------
+
+
+Tasks
+~~~~~
+// List what would need to be done to implement this Proposal in a few words:
+// * item ...
+
+
+Pros
+^^^^
+// add just a fact list/enumeration which make this suitable:
+//  * foo
+//  * bar ...
+
+
+
+Cons
+^^^^
+// fact list of the known/considered bad implications:
+
+
+
+Alternatives
+------------
+//alternatives: if possible explain/link alternatives and tell why they are not viable:
+
+
+
+Rationale
+---------
+//rationale: Describe why it should be done *this* way:
+
+
+
+//Conclusion
+//----------
+//conclusion: When approbate (this proposal becomes a Final) write some conclusions about its process:
+
+
+
+
+Comments
+--------
+//comments: append below
+
+
+//endof_comments: