History of OtherTuningSoftware/CommonInfrastructure

Older	Newer
		2005-02-10 23:37:55 . . . . MembersPage/RichardBarrington [GUI shouldn't be in common. Comms should.]

Changes by last author:

Added:

This page is for tuningsoftware developers

The common infrastructure will save lots of work.

Aim:

* separate data manipulation and GUI as well as possible

* make the tuning softwares very easy-to-configure (glade rulez)

* preferrably platform and language independent (glade rulez)

* let it be easy to could include a new parameter in a few minutes (without having to throw previous config files away)

----

Tools:

* glade is the right tool for defining gui (CVS firmware/doc/megatunix/Gui/genboard_knock.glade ) used in MegaTunix and PvTune, the perl based tuningsoftware

** Glade is very good, but GUI should stay out of common infrastructure. A PDA, Phone, or GBA GUI has different needs to a PC GUI.

* data mapping files (CVS firmware/doc/megatunix/Gui/genboard_knock.datamap ) is good for defining config variable mapping to gui elements. First appeared in MegaTunix (inspired by these pages), than slightly extended (see from bin/knock_gui_test.pl line 262). Data mapping resource is the key common denominator that all proper tuning software must use.

** currently text file is used (note: there could be defaults for variables; maybe some hierarchy or getting default from "ancestor" if otherwise not specified. This means 1)specified value; 2)specified ancestor 3)hierarchy ancestor 4)default value is the order of precedence)

** XML could be used of course

* Design tool (UML-Java modeling) : TogetherJ for Eclipse 3.0 from http://www.borland.com/ It seems to me that nicer than Rational rose. Maybe it worths to use it for OtherTuningSoftware/JTune so it gets as good as possible.

----

Not yet common - every implementation does it's own way

* communication low level

* parsing global.h config_t

* communication (variable-decode)

** This should be common. Data structures and manipulation should be common too.

Some of these will never be merged.

----

Beware: the rest is outdated, feel free to clean up

According to the plans, the minimum required functionality of any tuning software should be:

* interpret the layout file and provide a surface to the user for editing the parameters

* load and save config files (unknown meta-data and every data of unknown parameters should be saved too)

* upload config_t data to the board via RS232 or TCPIP

Also advised (but not absolutely necessary):

* import mcd files

* merge mcd files with config files

The "professional" tuning software is probably going to be written in Java (maybe an improved version of OtherTuningSoftware/JTune). This software should run on every platform, including PDAs, and it should also provide useful tools for advanced tuners (like log file handling, charts, histograms, etc). Other tuning softwares can be very useful (for example, a win-based sw for dumb users), and they should also implement these functions in order to be absolutely parameter-compatible.

----

header file: global.h

mcd file: a text file containing config_t numbers in numeric form together with their header file names (in "name=value" format). mcd files can be downloaded from the board

via serial connection and then loaded back into the tuning software

config_t numbers: the bits/bytes/words of the config_t structure in the header file

parameter: the humanized representation of config_t data, can be booleans/integers/real number

group: a set of parameters and/or other groups

----

The layout file consists of:

* configuration form layout (in XML format?), special tags for a given tuning software are allowed but they should work with a simple layout file, too

* parameter definition containing the parameter name, reference to the config_t numbers (by their name in the header file), and the data type specification

* parameter hierarchy (group definitions containing group name)

Data types can be defined and constructed from the following components:

* boolean or integer bit mapping/shifting

* linear or nonlinear two-way conversion formula (integer to real number)

* upper/lower limits (defaults are the binary limits of the config_t number, presuming that it's unsigned)

* unit specification

So a parameter can be either a boolean, an integer or a real number, determined by the data type definition. Parameter arrays (one or two-dimensional) can also be used similarly to single parameters (one name, dimension(s) of the array, reference to the config_t bytes, and one data type).

----

A config file contains, for each property:

* the property name

* its value (or values, if it's an array)

* relevancy / dependency (can inherit from parent group). Eg. if solenoid idle is configured, the properties only relevant to stepper should be hidden.

* confidence level: a real number that is showing how sure we are in the correctness of this value. The user can set this for him/herself, the default is -1, which means "never changed". The user can set a confidence level of 0..1 for each parameter.

* user comment

And for each group:

* the group name

* relevancy / dependency

* confidence level: if the confidence level of any descendant (parameter or sub-group) is undefined, it inherits the level of its ancestor group

* user comment

Note that the firmware doesn't care about confidence or comments, the user cares - absolutely:

The magic word is maintenance. When someone starts to make his configuration there are config files flying here and there

* noone knows what state it is, where variables came from, whom it was checked by, and how certain they are.

* having some comments make it somewhat better, but if we can make formal syntax (parsable by computer) is always better than text that can only be processed by IQ > 140

* there is only one thing that is even worse than the current config maintenance practice: if config cannot be exported to human readable and editable format, or cannot be carried across versions: this results in long and error-prone manual click here and there in some software instructions to achive some result (that is horror, resembles mswindows-land).

* we want to be able to highlight properties with low confidence level, since that means those were not verified with enough caution. To help the user, a filter (that can be turned off by selecting show all) can be helpful that hides very confident properties so the tuner can focus on the remaining stuff without using paper, memorizing it and other errorprone cavemen methods.

* link objects: groups and properties are linked under groups (or subtrees) and it's better to have a link object rather than have this information in the child. Eg. when a VE table consists of 8 kpa-lines, and each kpaline consists of 16 elements, the exact position of the child would be stored in the link object, not the child (and naturally not the parent). This allows several independent groupings if one likes, without overloading the child with linking-related information.

----

Natural view

The natural listing of properties is a tree browser. This is a direct view (and controller, as it allows changes too) on the model, very human friendly in this case. (The other view/controller is the mcd file, which is firmware friendly, and relatively close to the model, but not direct).

Eg. the config subtree would hold configuration parameters of an ECU. The leaves are human (rather than mcd) terms, eg. it can be an ON/OFF flag bit that is a separate property, that naturally maps to a bit of a byte in the mcd structure (the same byte would hold other flags too).

* config

** config.iac: idle control properties subtree

*** config.iac.pid - precise idle related (PID) parameters

*** config.iac.solenoid all solenoid related properties

There would be an other subtree, called runtime. This would hold entities fetched from engine operation. Most of these cannot be set, only viewed, and collect statistics. But some can be set as well.

* runtime

** runtime.sensors

*** runtime.sensors.wbo2

**** runtime.sensors.wbo2.ri - Ri measurement result

** runtime.debug

*** runtime.debug.ignition

**** runtime.debug.ignition.advance - this can be viewed, and it can be set (forcing a given advance, not used during daily drives normally :-).

Note that having a primary tree structure does not stop the user to make composite controllers and views in any combination. Just drop nodes to a form to collect what you want to see on one form.

The key is to have very simple properties in the leaves that are easy to present, explain and for the user to change, and for us to store the result.

When we have this, the rest is very simple: a composite cockpit (of different type properties os same type, like for the tables) is nothing more than a collection of several, simple views of simple individual properties. Note that the view of a property can be textual or graphical (like a gauge). Both can be packed to the same window if one wants.

----

This is only a rough draft yet, please add your comments.

Sounds mostly reasonable. A few points I've discovered so far:

* Need to ensure any concrete code for the GUI or IO interaction is pulled out. Java is close enough to cross-platform but the GUI, network and serial port access is not. Data/file access can be different on various platforms too (thinking PDA here). Limiting to interface definition might be the way to go here.

* KISS has to apply. If any PDA build is desired, a *lot* of the normal Java features taken for granted won't work. Even some things like the Number classes are missing.

* It should be possible to access all functionality for testing via JUnit.

Other thoughts:

* I'd like to see any firmware dependant files generated by the build process, as much as possible.

* On that note, we need to improve versioning information in the firmware. Would be nice to be able to query the firmware for it's capabilities, then configure tuning software to suit at runtime. We can kind of do that now by processing the likes of mct and mcd. (also check mdv command)

* I'd like to see an online repository of setups (and partial setups) available over a network interface. This suggests mapping of features to data blocks (I think this is the same what I call hierarchy of config properties : like VE related are grouped, etc..) which is already the case, but perhaps not well enough documented (no, grepping the sources doesn't count).

-- Rich.

----

First Java class definitions are in the CVS JTune module. Should they be interface definitions from the start, or is it OK to just later extract the interface from the classes when it's needed (to substitute alternative implementations) ? (eclipse helps extraction).

The plan is to model in java (only a tiny bit more work than UML or broken drawing tools, but it needn't be thrown away, maybe just refactored - we use eclipse so that should be easy).

The other logical choice would be defining in XML, as in the (brilliant!) entity engine of http://ofbiz.org . A 3d possibility would be SQL, so CMP EJBs can be generated from them.

the drawing is not here, but I scetch some thoughts from memory

* ConfigElementGroup : this could be recursive, so a tree is possible? That would be nice for browsing and managing

* ConfigElement : has matching

* ConfigElementSimple

* ConfigElementCompound: (maybe the big arrays can be double, and only one ConversionFormula)

* ConfigElementArray (maybe variable dim so same as matrix?)

* ConfigElementMatrix (it can have n x m ConfigElementSimple,(maybe just nxm double, and only one ConversionFormula )

* ConfigElementHelp (there can be several attached, with different languages)

* ConfigElementDepends (eg. on FirmwareFeature - or version, but as Richb said, an api for finding out supported feature is a must)

* ConversionFormula(SimpleConfigElement, FirmwareFeature -which has some version info, maybe the same data will be sent to ARM in a different format ) The formula can be described in reversed polish notation first, as noone yet came up with a simple offtheshelf interpreter that is easy to call from java or C.

Also check the XML format that http://www.deltasighk.org/miata/winlink/default.aspx uses for similar formulas.

ConfigMetadata can be attached to ConfigElementGroup, or ConfigElement (simple or compound) or an element in the matrix

* ConfigMetadata

** ConfigConfidence (when represented as double, 0.0 .. 1.0 is unknown to absolutely certain, -1.0 can be inherit parent). When importing a configtree with set confidence = 0.3 + 0.2*confidence_original, we apply the config properties that are higher or equal confidence than current - the one we are importing into - this can easily import settings from an alien configtree without screwing the config elements that we are already certain of.

** ConfigSource where the item came from

** ReviewedBy (obvious what it means)

When we have the [treebrowsing], and the display/set widget for changing the leaf types, we have a nice tuningsoftware. It seems to me we will need the tree structure even on PDAs: the offtheshelf swing implementation doesn't help there, but the child nodes can be displayed and reached with a click from a parent node.

Logging:

* we need a data serialization framework (using data-frames, metainfo, checksum) so anything can be logged. Either char or block device (RS232 / MMC).

We should make a nice module (that heavily uses histograms) that extracts and displays nice info from the log streams (wether they came from rs232 or file coming from mmc or internet).

----

Useful tools

* MVC (Model-View-Controller) concept; Richb started JTune this way

* Properties http://java.sun.com/docs/books/tutorial/javabeans/index.html and PropertyEditor

* JAXB (XML serialization; marshalling, unmarshalling) It's very likely that serialization (writing to file, sending via network) to XML (check JAXB, XPATH) is the way to go, of course serialization to the mcd and mtt format is also needed, but that'll be pie.

----

something between the widget and the physical layer

* obviously, we will have widgets (view) that can display information and user can use to control the given parameter (controller)

* .... something desired here .....

* we will have the conversion layer that use formulas to convert values, can read mcd format, write to mtt format etc...

* and the physical layer that can buffer the commands and send to ECU (connected serial or TCP/IP), write to local file or fetch from internet

Also something for navigation (a layout manager) among the large number of parameters, a tree-browser or some other way.

I knew for a long time something is missing from here. Now I know what was missing:

we need a simple object that knows about 2 values of a given parameter and implements sync strategy

* it knows the widget side value

* it knows the ECU side value

* it can autosync in any direction (observer pattern)

* it can sync on explicite request

* it can sync conditionally according to a rule. Rule can use input variables like confidence, values, value difference thresholds, maybe others. (??) The sync can be in some active communication with the tree-browser's filter (check the eclipse.org CVS plugin, team-synchronize view, that wonderful feature becomes automagically possible) so maybe only different config variables are shown, or node color can depend on equality.

The last capability allows huge flexibility:

* protection of ECU values (when someone is just browsing, no intent to change config); still very efficient when one knows what he's doing and selects autosync to ECU

* comparison of different configs (read one eg. from internet, read another from internet, file or ECU with no autosync)

* late writeback: delay writeback to ECU after several related variables are configured.

Some of this SyncManager functionality is needed anyway, it's better not to scatter it all around classes. If we have it separated, it becomes very powerful.

The beauty of it is that both sides around it (the widget and physical side) are simple and the SyncManager itself is also simple.

Note:

* there won't be separate infrastructure for loading and saving files. The exact same interface/methods will be used for files as when talking to the ECU. Just different concrete classes at the bottom, of course.

* Selective loading of variables or variable subtrees becomes automagically possible (just set the confidence for a subtree and there you go...)

* the same is used for configuration (that can be viewed or set) and runtime parameters (that can usually just be viewed, rarely set too - like the variables available from md. menu can only be set, not queried easily). The widgets can be different, of course, the widget has very little constraints (be sanely compatible with the given variable)

The SyncManager is a good place to hook observers, that can (for example) take logs or fire alarms when necessary.

----

everyone's homework: get a copy from the book "Design Patterns" (Erich Gamma, ..., ISBN 0-201-63361-2) from your tech library, read the

* "programming to interface, not implementation" chapter,

* application, toolkit, framework part

* and at least the abstract of patterns

You'll find it absolutely useful. Even though I was somewhat familiar with everything, seeing it written with a nice vocabulary will make sure we can talk about the design easily and makes it easy to find if we are on the right track at a given point.

If you have an electronic file of the book or the given chapters, link here. There is a DesignPatterns wiki somewhere also.

----

== Configuration As a Tree ==

In order to come up with a clean way of understanding/describing the interface to the genboard, I've been reading MegaTunix's MSNS datamaps. The following is a really rough draft of organizing the genboard's interface as a tree:

: > <code>

</table>

</table>

<fuel>

<! ~~add vital fuel config info here.~~ >

</priming>

</table>

</cranking>

</table>

</warmup>

<accel>

</tpstrigthresh>

</accelenrichdur>

</coldenrichaddon>

</coldenrichmult>

</table>

<decel>

</decel>

</accel>

<ego>

</egoactivetemp>

</egoactiverpm>

</egoswitchingvoltage>

</egostep>

</egoignevents>

</egolimit>

</ego>

</enrichment>

<ve>

</table>

</ve>

</fuel>

<! ~~add ignition parameters here~~ >

</table>

</trimangle>

</fixedangle>

</advance>

</ignition>

</code>

I've used XML/HTML/SGML-like syntax, because it's something most people understand. I don't particularly care for it; the important part is representing things as a tree.

The other thing that's different is the way I'm representing the conversion step. I'm using postfix notation instead of traditional infix notation (every day algebra). This is because it's easier to parse/execute postfix (using a simple stack) than infix.

-Alexander

----

libraries we are likely to use

though under evaluation currently

OtherTuningSoftware/CommonInfrastructure/Abeans

----

interesting programs to get ideas from

* http://kgb.ijs.si/KGB/Files/ICALEPCS2003-CS_Comparison.pdf Comparison table of control systems: abeans looks compelling

* check what they use for syncrotrons control. Somewhat more complex than GenBoard, but same type of control tasks are needed:

** http://www.cosylab.com/page.php?mid=3&sid=34 (links at the bottom!!!)

** http://kgb.ijs.si/demos/

* run the following command: appletviewer http://kgb.ijs.si/demos/SVApplet.html?hem_id_l=5&hes_id_l=29

Search: