ANN: kbmMW Professional and Enterprise Edition v. 5.03.00 released!

We are happy to announce v5.03 of our popular middleware for Delphi and C++Builder.

If you like kbmMW, please let others know! Share the word!

We strive hard to ensure kbmMW continues to set the bar for what an n-tier product must be capable of in the real world!

This release is fairly large and brings several new features:

  • Automatic database upgrade to match your ORM classes (please test carefully before deploying to production environments containing existing data!)
  • New! Template based binary parser for advanced bit fiddling
  • New! Detailed metadata extraction and creation on many supported databases, including SQLite, MSSQL, Oracle, PostgreSQL, MySQL, Interbase, Firebird
  • New! Virtual fields and tables in ORM
  • New! Thread safe configuration system for easy one style read/write access to configuration files and registry.
  • New! Debug visualizers for kbmMWNullable and kbmMWDateTime values.
  • New! CHM help files with 24800+ topics.
  • + Many many more additions and improvements and bugfixes for reported bugs

Please look in the end of this post for a detailed change list.

Professional and Enterprise Edition is available for all with a current active SAU. If your SAU has run out, please visit our shop to extend it with another 12 months.

CodeGear Edition is available for free, but only supports a specific Delphi/Win32 SKU, contains a limited feature set and do not include source.

Please visit to download.


kbmMW is the premiere n-tier product for Delphi, C++Builder and FPC on .Net, Win32, Win64, Linux, Java, PHP, Android, IOS, embedded devices, websites, mainframes and more.

Please visit for more information about kbmMW.


Components4Developers is a company established in 1999 with the purpose of providing high quality development tools for developers and enterprises. The primary focus is on SOA, EAI and systems integration via our flagship product kbmMW.

kbmMW is a portable, highly scalable, high end application server and enterprise architecture integration (EAI) development framework for Win32, ..Net and Linux with clients residing on Win32, .Net, Linux, Unix, Mainframes, Minis, Embedded and many other places. It is currently used as the backbone in hundreds of central systems, in
hospitals, courts, private, industries, offshore industry, finance, telecom, governements, schools, laboratories, rentals, culture institutions, FDA approved medical devices, military and more.


5.03.00 Sep 16 2017

Important notes (changes that may break existing code)
* Fixed Delphi C++ HPP generator errors related to kbmMWNullable.pas and kbmMWDateTime.pas
The changes needed to fix it for kbmMWNullable.pas unfortunately require
us to default unsupporting assigning of a variant directly to a kbmMWNullable.
However if you are using Delphi ONLY and will never use the C++Builder header files
you can add {$DEFINE KBMMW_DELPHI_ONLY} in
Then you can still assign a variant directly to a kbmMWNullable in Delphi.
The changes in kbmMWDateTime is using overloaded versions of methods
instead of using default value Math.NaN which is translated incorrectly
by the HPP emitter.
* Removed PrependTableName from all metadata classes and thus it may
error with missing property on opening forms/datamodules containing
metadata components. Ignore that specific property missing error and save.

New stuff
– Added support for defining root type for marshalled objects. Array or Object.
– Added support for using nth next best marshalling type if primary is not registered in known types.
– Added log to AutoRegisterServices in kbmMWServer.pas
– Added support for bit values in YAML. A bit value starts with b or B and contains only 0 or 1’s
– Added support for default metadata class for database adapters. It will be used when
no explicit metadata component has been linked to the connection pool.
– Added OnLoginSuccess and OnLoginFail events to TkbmMWAuthorizationManager.
– Added support for understanding bit values in TkbmMWObjectNotation.
– Added Path method to TkbmMWONCustomObject which will take a delimited path and
traverse the tree optionally adding needed objects on the way.
– Added support for custom properties and basic properties in AMQP.
– Added support for Default date/time in TkbmMWDateTime.
– Added Reset method to TkbmMWDateTime, completely clearing its value.
– Added additional constructors to TkbmMWDateTime allowing direct timezone information.
– Added Memory2Binary and a number of BitCount methods to TkbmMWPlatformMarshal.
– Added support for reference counted data in TkbmMWAutoValue.
– Added kbmMWGetApplicationVersionInfoAsString and kbmMWBackupFile global methods to kbmMWGlobal.pas.
– Added support for specifying quote in kbmMWQuoteString and kbmMWUnquoteString.
– Added support for different start and endquote in kbmMWImplode.
– Added kbmMWDigitCount2Multiplier, kbmMWUnsignedDigitCount, kbmMWSignedDigitCount and
kbmMWExplodeFloat to kbmMWGlobal.pas, which determines metadata of a float value.
– Added kbmMWBitStringToInteger and kbmMWIntegerToBitString to kbmMWGlobal.pas
– Added overloaded Get method to TkbmMWSmartClientORM requesting dataset from server,
returned as an object/list, and providing custom arguments.
– Added overloaded RunSQL methods to TkbmMWMemSQL allowing executing SQL directly on
one or more provided TDataset sources.
– Added support for multistatement execution in TkbmMWMemSQL. Statements are separated by semicolon.
– Added support for salting initial string or byte keys in kbmMWCipherHash.pas.
– Added Clone method to TkbmMWCustomDef in kbmMWConnectionPool.pas
– Added support for detecting deleted fields in TkbmMWFieldDefs.
– Added MetaGetDef and MetaRename functions to TkbmMWCustomMetaData.
– Added significant changes to TkbmMWCustomSQLRewriter, including datatype mapping
and rewriting of numerous statement types.
– Added DefaultMetaData and CurrentMetaData properties to TkbmMWCustomConnectionPool to
allow for using kbmMW specified default metadata, alternative specific assigned on.
– Added support for piping outcome of query thru internal SQL statement as postprocessing
in TkbmMWCustomPooledDataset. Syntax: somequery <!PIPE!> somepipestatement
somequery is the query sent to the backend database. The resultset from that
is the piped thri the kbmMW SQL type somepipestatement.
Use constant KBMMW_QUERYPIPE to add pipe statement.
– Added support for mwfdctStrictPrecision and mwfdctStrictSize in TkbmMWFieldDefs.ChangedFields.
Strict indicates values must match exactly.
– Added DeletedFields, DeleteByFieldName and GetByFieldName to TkbmMWFieldDefs.
– Added additional overloaded Rewrite methods to TkbmMWCustomSQLRewriter.
– Added Terminate to TkbmMWScheduledEvent to prematurely terminate ongoing execution.
– Added Delay method to IkbmMWScheduledEvent which can be called at any time
including during execution of the said event to add additional delay to the
coming run’s of the event.
– Added support for multistatements in kbmMWSQLite.pas
– Vastly expanded TkbmMWCustomSQLMetaData.
– Added support for readonly and virtual field attribute in TkbmMW_FieldAttribute (kbmMW_Field)
– Added new TkbmMW_VirtualFieldAttribute (kbmMW_VirtualField) to indicate field
which will show up in resultset but do not exist in datastore.
– Added CreateDBDataset to TkbmMWORM which will generate a TDataset
matching the referenced TkbmMWORMTable.
– Added CompatibleTable, GetUpgradeTableStatements, UpgradeTable to TkbmMWORM which
are used for checking if a class is compatible with the underlying datastore,
and which can either directly upgrade the underlying datastore or suggest
statements to make the upgrade happen. Use with caution on production systems!
– Vastly expanded kbmSQLRewriter.pas to support metadata, indexes, new statements etc.
– Added Reset method to kbmMWNullable to reset its contents.
– Added PrettyChar to TkbmMWCustomLogFormatter to be able to selectively
choose what each byte value should display in the log.
– Added additional Warning, Error and Fatal overloaded methods to TkbmMWLog.
– Added support for property AutoSplitLines in TkbmMWStringsLogManager.
– Added support for providing HTTP related values (mwatURL, mwatHost, mwatUserAgent,
mwatHeader (named item), mwatCookie (named item) as argument in smart services.
– Added support for new kbmMWConfiguration unit which provides access to automatically
load and optionally save configurations to various storages.
Either use Config.As…. methods directly, or specify attribute
[kbmMW_Config(‘path:a.b.c.value, direction:rw’)] on any
property or field in a class.
If the class descends from TkbmMWConfigurableObject, the field/property
will automatically receive the value as defined in the Config object,
when the class is instantiated, and similarly store the value back
to the Config object when the class instance is destroyed.
Classes not descending from TkbmMWConfigurableObject can also
take advantage of the attribute. Just make sure to
call Config.ReadConfig(TValue.From(self)); or Config.WriteConfig(TValue.From(self));
for example in AfterContruction and BeforeDestruction methods of your class.
Also register your class as a kbmMW known class.
– Added support for using kbmMW_Config attribute for arguments in smart services.
– Added new unit kbmMWBinaryParser which can be used for parsing binary data
according to a custom template. Check the blogs on
for info.
– Added new debug visualizers for kbmMWDateTime, kbmMWNullable,
kbmMWDuration and TkbmMWAutoValue values during debug.
– Added Object Notation based stats (kbmMWONStats) unit for
registering statistics in object notation format, which
hence can be saved and loaded as any of the object notation
streaming formats.

– Fixed scheduler so exceptions will not stop execution threads.
– Fixed incorrect Args[0] in REST transport stream. Now correctly contains called URL.
– Fixed detection of added and changed fields in TkbmMWFieldsList, and added detailed info about differences.
– Fixed XML namespace crash issue.
– Fixed REST streamformat incorrectly double translating UTF8
– Fixed deletemessage and rejectmessage when provinding a message already tentatively
popped resulted in message not being deleted/rejected, but instead leaked.
– Fixed TkbmMWDynamicLockFreeHashArray and siblings related to deadlock (due to using
locking without escalation support for performance reasons) and due to incorrect
management of capacity of GC generations.
– Fixed potential leaks happening on exceptions in certain places in kbmMWORM.
– Fixed occational A/V due to reporting triggering memory logging.
– Fixed REST handling of returned objects (bug introduced in 5.0.2)
– Fixed DOA adapter related to LOB
– Fixed TkbmMWStringBuilder.GetLastChar which amongst others affected YAML parsing.
– Fixed scheduler clean shutdown of defined events when removing them.
– Fixed initialdelay not respected in Scheduler.
– Fixed destruction shutdown bugs in Scheduler.
– Fixed YAML null parsing bug (hazeltine).
– Fixed various parsing bugs in YAML.
– Fixed time to execute calculation in kbmMWScheduler.pas.
– Fixed string offset issues for Linux.
– Fixed IOS TkbmMWTiming timebase issues.
– Fixed A/V bug in TkbmMWDOMXMLNameSpaceList during destruction, due to
non owned namespaces.
– Fixed IBO adapter compile errors.
– Fixed Delphi C++ HPP generator errors related to kbmMWNullable.pas and kbmMWDateTime.pas
The changes needed to fix it for kbmMWNullable.pas unfortunately require
us to default unsupporting assigning of a variant directly to a kbmMWNullable.
However if you are using Delphi ONLY and will never use the C++Builder header files
you can add {$DEFINE KBMMW_DELPHI_ONLY} in
Then you can still assign a variant directly to a kbmMWNullable in Delphi.
The changes in kbmMWDateTime is using overloaded versions of methods
instead of using default value Math.NaN which is translated incorrectly
by the HPP emitter.
– Removed PrependTableName from all metadata classes and thus it may
error with missing property on opening forms/datamodules containing
metadata components. Ignore that specific property missing error and save.
– Fixed some TkbmMWORM.ExistsTable overloads incorrectly calling CreateTable.
– Fixed TkbmMWORM.ListFromDataset returning undefined value upon error.
– Added UTCAsVariant, LocalAsVariant, LocalDateAsVariant and LocalTimeAsVariant
properties to TkbmMWDateTime.
– Fixed warning at installation of designtime package related to smart service setup.

Changes/minor additions
– Improved shutdown of transport, and scheduled events.
– Checked encryption and updated SQLite Server/Client demoes to support compression.
– Updated AMQPClient demo with sending with properties sample.
– Set AnonymousResult to default true for REST attributes.
– Improved kbmMWGetFileVersionInfoAsString to also support Android.
– Improved server side ForwardMessage, ForwardRequest, ForwardResponse when original message comes from client messaging transport.
– Improved client transport disconnect.
– Updated Metaxxxx methods on database adapters to take AOwner argument.
Typically it will be empty, but with index relations, it will hold table name.
– Improved support for providing NodeID in server side messaging transport, even
though contents is encrypted.
– Cleaned up kbmMWGlobal.pas by grouping all kbmMWTextxxxx methods as class methods in TkbmMWTextPChar,
and added similar set handling bytes as class methods in TkbmMWTextPByte.
– Improved kbmMWGetComputerName and kbmMWGetComputerModel to support additional platforms.
– Changed Save and Load methods of TkbmMWCustomStat to not take any field names.
Specialized versions of TkbmMWCustomStat will handle that.
– Improved REST streamformat marshalling.
– Improved TkbmMWClientTransactionResolver.
– Renamed TkbmMWCustomRewriter to TkbmMWCustomSQLRewriter.
– Improved stability of event execution in kbmMWScheduler.pas during exceptions.
– Optimized TkbmMWLockFreeHashArray to use generational garbage collection for
higher performance and concurrency.
– Starting to converting some utility threads (garbage collection and such) to
use be kbmMWScheduledEvent instead.
– Improved object marshalling.
– Changed default log level filter in TkbmMWCustomLogManager to [mwllBasic,mwllNormal].
– Changed default flush value in TkbmMWStreamLogManager to 0 (force each write to disk)
– Improvements added to kbmMWRTTI.pas
– Improved kbmMWDebugMemory.pas to detect when reporting is happening to avoid deadlocks.
– Enhanced TkbmMW_ArgAttribute (kbmMW_Arg) to allow for referencing specific argument index alternatively specific named argument.

REST easy with kbmMW #7 – Configuration

The upcoming release of kbmMW will also contain a brand new configuration framework.

What is a configuration framework?

You have probably already used one many times, in the form of reading or writing data to/from INI files, registry settings, your own XML files etc.

Basically most applications needs to obtain some user defined configuration from somewhere.

In kbmMW we have until now been doing it the same way as everybody else, but it is so boring and tedious to handle configurations the old fashioned way.

Why do we have to write all that boiler plate code to read some data form a configuration, or to store something back into the configuration, and doing that we usually forget doing backups and such, risking to damage a working configuration, if we experience an unexpected power failure at the wrong time.

But that is all past now 🙂


kbmMW’s new TkbmMWConfiguration class and it’s accompanying storage classes descending from TkbmMWCustomConfigurationStorage will automate all the tedious work for you.

So how to use it?

The simplest way possible to read a configuration:

unit Unit1;



procedure TYourClass.DoSomething;

As you can see, you will have a thread safe Config instance readily available for you, and we are using it to read some string out, and store a string back.

So where is this configuration then persisted?

It will default create and access an XML file named yourapplicationname_config.xml placed in the startup directory of your application.

It’s contents, given the above example, would look like this:

<?xml version="1.0" encoding="utf-8" ?>

As you can see, it will use the dot notation of your keys to allow you to group your configuration settings anyway you want.

Obviously there are several additional methods to AsString which can be used for fetching or storing other types of data.

Currently kbmMW provides:

  • AsInt32
  • AsInt64
  • AsDouble
  • AsBoolean
  • AsDateTime
  • AsBinary
  • AsString

And some functions to easily return a default value, in case nothing is defined in the configuration file. They all follow the same syntax like this one:

 function AsDefString(const APath:string; 
                      const ADefault:string; 
                      const ASetIfDefault:boolean = false):string;

Notice the ASetIfDefault value. It is default false, but if you set it to true, the configuration storage will be updated with the default value, if no value is found for the key given by APath.

The other functions are

  • AsDefInt32
  • AsDefInt64
  • AsDefDouble
  • AsDefBoolean
  • AsDefDateTime
  • AsDefBinary

You can test for if a value exists in the configuration file by using the contains property:

if Config.Contains['myconfig.somepath'] then...

The configuration file is automatically opened and loaded the first time you use any of the above properties/functions.

Notice the distinction between opened and loaded! If you remember the good old INI file, then you open it, and every time you read from it, you will read from the actual file, and thus immediately see changes without the need to reopen it. The same when using the Windows registry.

That may be a nice thing, or it may be wasted CPU cycles depending on the scenario. If you read a value lots of times, you would usually put the configuration into a variable that you manage, not to waste too much energy reading it from the INI/Registry.

Configuration storages

How does kbmMW do it? It depends on which of the configuration storage’s that is being used.

kbmMW currently supports the following storage’s:

  • TkbmMWIniConfigurationStorage
  • TkbmMWRegistryConfigurationStorage
  • TkbmMWJSONConfigurationStorage
  • TkbmMWXMLConfigurationStorage
  • TkbmMWYAMLConfigurationStorage

The Ini and Registry configuration storage operates live on the INI file and the Registry entries, as you would normally expect.

The remaining configuration storage’s operates on an internal representation of the configuration, which will be streamed back into the file at relevant times.

What is a relevant time?

  • Upon application shutdown if anything has been changed
  • Upon calling Save method on the storage if anything has been changed
  • On every change, if AutoSave property is true

If you use the XML, YAML or JSON storage, you will even have the advantage of kbmMW automatically backing up your old configuration before saving the new.

You can control how many backup files you want to leave via the BackupMaxCount (default 5) property, and where it should be backed up to via the BackupPath (default same path as original file) property and finally what extension the backup file should have via the BackupExt (default ‘bak’) property.

The XML, YAML and JSON storage methods all stores the data in a tree structure following the dotted names of the given path.

The Ini storage method use the first segment of the path as the section name, and the remaining part of the path (including dots) as the key name. Eg.


Will result in an ini file like this


The Registry storage method use the last segment of the path as the name value and the remaining as the registry key tree path. In addition one specify the RootKey (defining which registry hive to operate) and the starting place in the tree within the hive when creating an instance of the storage.

If you want to change the storage method of the configuration, you do like this before using any of the access methods:


With the above registry storage, setting Config.AsInt32[‘a.b.c.def’]:=10 would result in opening the registry key Software/MyCompany/a/b/c in the registry hive HKEY_LOCAL_MACHINE, and setting the value def to the integer value 10.

kbmMW automatically opens and closes registry keys as needed.

Split configuration

You have seen how easy it is to access the standard Config object. But what if you explicitly want two (or more) different configurations concurrently?

Solution. Instantiate your own TkbmMWConfiguration instances. Eg.

  myConfig2.Storage:=TkbmMWXMLConfigurationStorage.Create('myconfiguration2'); ...

This way you can also specify your own naming of the configuration, which in the above examples results in myconfiguration1.xml and myconfiguration2.xml unless you also decide to change the configuration storage method.

myConfig1 and myConfig2 can then be used the same way as we used the standard Config object.

Automatic configuration

It is now already simple to access the configuration from anywhere in your application, but you can also make it automatic.

Say you have an object which you want to hold certain values based on your configuration. You could yourself instantiate the object and assign the values using the As… properties, but there is also another way.

TMyObject = class(TkbmMWConfigurableObject)
  property SomePath:string read FSomePath;


By descending your object from TkbmMWConfigurableObject, and using the kbmMW_Config attribute on either properties or fields, your configuration will automatically be read and optionally written when you instantiate or destroy your object.

Upon instantiating TMyObject, then the property SomePath would have the value ‘SomeNewPath’, provided this blog previous examples were followed.

Default the fields/properties with the attribute on are only read from the configuration storage. If you want the configuration to be automatically updated with new values from your object, then you need to provide an additional argument to the attribute:


You can also define if date/time values are read and stored as UTC (default) or local time values, by setting yet another argument.


kbmMW also supports FSomeTime being a regular TDateTime.

You can force the reload of the configured properties by calling the
ReadConfig method of your configuration object, and you can force saving the properties that supports being saved by calling WriteConfig.

You can switch which configuration to load the configuration from by setting the Configuration property of your object to point to a relevant kbmMW configuration instance and explicitly call ReadConfig.

What if you already have an object hierarchy and it is not possible for you to inherit from TkbmMWConfigurableObject? Then you can still use kbmMW’s smart configuration by manually requesting the configuration and registering your object as a known object with kbmMW. Eg. now not descending from TkbmMWConfigurableObject:

TMyObject = class
  property SomePath:string read FSomePath;


To load the configuration do:


and saving it

If you want to make it automatic for your own object, override the methods AfterConstruction and BeforeDestruction of your object class.

  procedure AfterConstruction; override;
  procedure BeforeDestruction; override;

And call the above ReadConfig and WriteConfig calls.

Smart services

Finally, if you use kbmMW smart services, you can also automatically use kbmMW configuration data as arguments for the service calls. Eg.

 [kbmMW_Rest('method:get, path: "echoreversedconfigstring"')]
 // This method obtains its value from the configuration.
 function ReverseStringFromConfig([kbmMW_Config('a.b.c.value')] const AString:string):string;

Upon calling the echoreversedconfigstring method it’s AString argument will always contain a string value read from the configuration.

This concludes this blog about the new configuration features in the upcoming kbmMW release.



REST easy with kbmMW #6 – Database 2

Now our fine REST server has been running for some time, and we start to understand we need to expand it with some more data.

Adding an additional table for new info is easy, as its done the same way as shown in REST easy with kbmMW #2 – Database

However what if we need to add additional fields to the TContact class? What do we do with the data storage?

Until upcoming release of kbmMW, we would have had to make our own table update mechanism, which often is easy to do, as long as you add new fields, and those fields should not be part of primary keys and such.

But from next release of kbmMW, we also release a new beta feature in the kbmMW ORM. The ability to determine if the data storage is compatible with the class, and the ability to automatically update the data storage to match the new look of the class.

It sounds so deceptively simple to do so, but when we also want it to work across different databases, handling indexes and more, it suddenly starts to require quite detailed understanding of databases and their metadata.

So as a spin off of this new feature, kbmMW also comes with vastly improved database metadata handling, even better understanding of SQL query variants, more features in our in house SQL parser and much more.

Ok.. enough of the sales talk…. How do you do it then?

The original TContact class was defined like this:

unit Unit9;




TContact = class
 [kbmMW_Field('primary:true, generator:shortGuid',ftString,40)]
 property ID:kbmMWNullable read FID write FID;

 property Name:kbmMWNullable read FName write FName;

 property Address:kbmMWNullable read FAddress write FAddress;

 property ZipCode:kbmMWNullable read FZipCode write FZipCode;

 property City:kbmMWNullable read FCity write FCity;

 property Comments:kbmMWNullable read FComments write FComments;



Lets add a Gender field, change the Name field to be unique (just for fun), and change the zip code field to be a string type matching the property type (previously we, perhaps incorrectly, defined it as an integer data storage field, where storing it as a string might have been better).

unit Unit9;




TContact = class
 [kbmMW_Field('primary:true, generator:shortGuid',ftString,40)]
 property ID:kbmMWNullable read FID write FID;

 [kbmMW_Field('name:name, unique:true',ftString,50)] 
 property Name:kbmMWNullable read FName write FName;

 property Address:kbmMWNullable read FAddress write FAddress;

 property ZipCode:kbmMWNullable read FZipCode write FZipCode;

 property City:kbmMWNullable read FCity write FCity;

 property Gender:kbmMWNullable read FGender write FGender;

 property Comments:kbmMWNullable read FComments write FComments;



Usually these changes in the class would render that class incompatible with the data storage. In fact we can now ask the ORM if the data storage is compatible with the class we have.

if not FORM.CompatibleTable(TContact) then
  raise Exception.Create('TContact is not compatible with the datastore');

When we call CompatibleTable this way, it compares using the strictest comparison method, which means that storage fields and index definitions must be not only compatible, but identical.

By adding an TkbmMWONObject instance to the call, we can be told what problems there are:

  if not FORM.CompatibleTable(TContact,issues) then
   // Decipher issues object.
   // There may be 3 properties in the object, named add, modify, delete
   // and each of those will be an array of objects with properties for name, unique, primary, required, size, precision and dataType.

This way you get detailed information about the changes needed to make the data storage compatible with your class.

It is possible to tune exactly what to compare and how, and thus limit the strictness of the comparison mechanism. This is done by adding one additional argument to CompatibleTable, namely the ACompatibleFlags:TkbmMWORMCompatibleTableFlags.

TkbmMWORMCompatibleTableFlags is a set of flags including:

  • mwoctfBasic – Basic strictness. Translates to mwfdctDataType, mwfdctPrecision, mwfdctSize
  • mwoctfStrict – Strongest strictness. Translates to mwfdctPrimary, mwfdctUnique, mwfdctRequired, mwfdctDataType, mwfdctStrictPrecision, mwfdctStrictSize
  • mwoctfConstraints – Include constaints validation. Translates to mwfdctPrimary, mwfdctUnique, mwfdctRequired
  • mwoctfType – Include generic data type validation. Translates to mwfdctSize, mwfdctPrecision, mwfdctDataType.

The translated comparing flags cant be provided directly, but is used internally, and only shown for completeness.

  • mwfdctPrimary – Primary key definition must match.
  • mwfdctUnique – Unique field constraint must match.
  • mwfdctRequired – Required field constraint (not null) must match.
  • mwfdctDataType – Exact data type must match.
  • mwfdctStrictPrecision – Field precision must match exactly. If not specified the data storage may have a larger field precision than required.
  • mwfdctStrictSize – Field size must match exactly. If not specified the data storage may have a larger field size than required.

Walking thru the issues table can be fun, but even more fun would be not to have to do so.

  // It will make a strict comparison (arguments can be added to choose non strict comparison),
  // and generate a list of statements in generic kbmMemTable SQL format
  // that can be used to transform the data storage to be compatible with
  // the class.
  // It could be ALTER TABLE xxx DROP COLUMN yyy

Now the observant reader may say: “Thats all fine, but I for a fact know that SQLite do not support ALTER TABLE DROP COLUMN statements! So it wont work!”

You are right…. about the SQLite limitation. However remember that kbmMW will translate the statements into something acceptable by the target database type, so SQLite will in fact suddenly be able to have a column dropped from a table containing data. kbmMW will do its best to make it happen.

If you would like to see the rewritten SQL. In other words generic kbmMemTable SQL converted to specific target database syntax, then do like this:

  // Now the list of statements will have been converted to 
  // the specific target database.

And after all this gibberish then how to make the data store compatible with the new class?


After running this, the table “contact” in the database will have been made compatible with the class, with all remaining data retained.

If you have huge tables with billions of rows, then it might be better to get inspiration from the output from GetUpgradeTableStatements, and apply the changes under human supervision.

Although kbmMW attempts to do things in a safe way, I also recommends backing up the data storage before attempting an automatic upgrade.

Currently kbmMW contains SQL rewriters that targets  SQLite, MySQL/MariaDB, PostgreSQL, MSSQL 2008/2012+, Oracle 9+,  Interbase/Firebird and generic SQL92 and SQL2003 compatible databases.

When the beta of this upgrade mechanism is released we urge people to test it upgrading capabilities carefully before deploying to production.

best regards


Encryption of a string

kbmMW Professional and Enterprise edition contains multiple cipher and hash algorithms. This short blog post shows how to use them to encrypt a string or binary data.

kbmMW bundles AES/Rijndael, Twofish, Blowfish, Misty, RC2, RC4, RC5, RC6, Tea, Serpent, Mars, IDEA, ICE, DES, 3DES, CAST128 and CAST256 encryption algorithms.

The currently most widely used algorithm around is AES so lets use that to encrypt a string.

function EncryptValue(const AValue:string; 
  const AKey:string):string;

It is as simple as that.

However there are many options one can choose for how the encryption shall take place, amongst others, how the string key should be converted to something that can be used for encryption, and because all encryption algorithms works on blocks of bytes, how UTF16 (unicode) strings should be converted to bytes upon encryption.

The InitString method defineds what key to use (in this case a Unicode string that is automatically converted to UTF8 before use, and that is then hashed using the SHA256 hashing algorithm, which is the defacto standard way to prepare keys for general AES encryption.

However to make things harder to break, you could use a different hashing algorithm, because most encryption breakers assume that your key is hashed using SHA256. Just make sure to use another hashing method that also outputs 256 bit keys and that is considered a strong hash.

However a better way to confuse encryption breaking, is to add a salt to your key.

The salt is a bit of information that is only known by you and the other end. It can be a machine ID, a pin code or something else that both sides knows about.

So if your key would be THIS IS MY WEAK KEY, you could salt it which would add some technical stuff to the start of it, making it more difficult to break, because now two things needs to be known to be able to guess your key.

A salt is typically an application and installation specific thing, so one instance of an installed server application will have its own unique salt, that all its clients also needs to know about. You could even give each client his own personal salt, as long as you are able to manage that on the server.

A stupid example could be to salt with the external IP address a client is approaching the server with.

Then when the user logs in via one client, the data will be encrypted differently to when the client logs in via another client.

kbmMW supports adding salt to a key via the InitString/InitBytes methods.


Packaging files securely using TkbmMWLookupStreamStorage

Chinese blog by Xalion about how to use the kbmMW virtual filesystem called TkbmMWStreamStorage and its descendant TkbmMWLookupStreamStorage in combination with TkbmMWCrypt to pack multiple files/streams into a single searchable and encrypted file.

English Google translation: Translate

kbmMW Binary Parser #1

A universal binary parser made available as part of kbmMW Enterprise Edition

Next version of kbmMW Enterprise Edition will include a definition file based (for the moment, fixed record) binary parser.

What does it do? Well.. it parses well formed binary (or textual) streams to extract telegrams and their contents. It functionality wise can be compared to a regular expression, just for bit and byte level information, although with simple scripting and calculation capabilities.

A telegram is in this sense a fixed length bunch of bytes, which may contain bit fields or byte fields or ASCII type string data.

The definition file defines how the telegrams are looking, what subparts they consist of, and what to do when a matching part has been found.

The outcome is typically a match along with a number of keys/values, or a failure to match with anything. The actual naming and use of the keys and their values is up to the developer to decide.

A definition file is default written in YAML and consists of 3 main sections:

  • TAGS

The VALUES section can contain a list of predefined values to be available before any attempt to match anything. This can for example be used for defining “constants” which your application understands or default values.

The TELEGRAMS section contains an array of the telegram masks to look for, and the TAGS section contains an optional number of named sub parts referenced from either the TELEGRAMS section or from within the TAGS section.

It may seem a bit vague right now, but it probably makes more sense when I show a sample in a moment.

The TELEGRAMS section and the TAGS section both contains masks and optional expressions to be executed when a mask match. They also both define if the mask is a bytes mask, a bits mask or a string mask.

Masks which has been defined as bytes masks, always operates on the byte level. Similarly masks which has been defined as bits masks, always operates on the bit level (currently maximum 8 bit per mask).

String masks are similar to bytes masks, except that they compare ASCII strings.

Let us look at a sample definition file. As YAML actively is using indentation to determine if something belongs to current definition or is a new definition, it is of high importance that the indentation is correct. YAML also recognizes lines starting with – as an entry in an array, unless the dash seems to be part of another property. In fact YAML is pretty complex in what it understands, but it does read easier for the human eye, why I chose it as the default definition file format.

The sample definition file is for a standard scale format called Toledo deviced by a company called Mettler Toledo many years ago.

YAML wise, the document actually contains an object with one single property named TOLEDO, which has 3 properties, VALUES, TELEGRAMS and TAGS.
The VALUES property has a number of properties with values. The TELEGRAMS object has one property with an array of objects each having a mask and optional expr property.

The TAGS property is an object which has a number of properties (SWA, SWB, SWC, DP etc) which each are objects containing a property named bytes/bits/string which is an object containing either a single mask and optional expr property, or an array of such.

It may take a while getting used to read and write YAML documents, but perseverance makes experts.

Lines starting with # are comments.

# This is a sample file showing how to parse Toledo telegrams
# using kbmMW Binary Parser

        # Unit constants
        C_UNIT_GRAM:                 2000
        C_UNIT_UK_POUND:             2001
        C_UNIT_KILOGRAM:             2002
        C_UNIT_METRIC_TON:           2003
        C_UNIT_OUNCE:                2004
        C_UNIT_TROY_OUNCE:           2005
        C_UNIT_PENNY_WEIGHT:         2006
        C_UNIT_UK_TON:               2007
        C_UNIT_CUSTOM:               2008

        # Status constants
        C_STATUS_OK:                 1000
        C_STATUS_DATA_ERROR:         1001
        C_STATUS_SCALE_ERROR:        1002
        C_STATUS_SCALE_OVERLOAD:     1003
        C_STATUS_IN_MOTION:          1004
        C_STATUS_TARE_ERROR:         1005

        # Tare constants
        C_TARE_PRESET:               3000
        C_TARE_AUTO:                 3001
        C_TARE_NONE:                 3002

        # Default values
        STATUS:                    @C_STATUS_OK
        TARE:                      0
        GROSS:                     0
        NET:                       0
        INCREMENT_SIZE:            1
        IS_POWER_NOT_ZEROED:       false
        IS_SETTLED:                false
        IS_OVERLOAD:               false
        IS_NEGATIVE:               false
        IS_CHECKSUM_OK:            false
        WEIGHT_FACTOR:             1
        TARE_FACTOR:               1
        TARE_CODE:                 @C_TARE_NONE
        TERMINAL_NO:               0
        WEIGHT_UNIT:               @C_UNIT_KILOGRAM
        TARE_UNIT:                 @C_UNIT_KILOGRAM

              - mask: [ 0x2, @SWA, @SWB, @SWC, 6*@W, 6*@T, 0xD, @CHK ]
                      - "TARE_UNIT=WEIGHT_UNIT"
                      - "GROSS=IF(IS_NETTO=0,WEIGHT,0)"
                      - "NET=IF(IS_NETTO=1,WEIGHT,0)"


                # bit offset 0
                mask: [ 0, 0, 1, 2*@IS, 3*@DP ] 

        # bit offset 0, 3 bits
              - mask: [ 0, 0, 0 ]
                expr: [ WEIGHT_FACTOR=100, TARE_FACTOR=100 ]
              - mask: [ 0, 0, 1 ]
                expr: [ WEIGHT_FACTOR=10, TARE_FACTOR=10 ]
              - mask: [ 0, 1, 0 ]
                expr: [ WEIGHT_FACTOR=1, TARE_FACTOR=1 ]
              - mask: [ 0, 1, 1 ]
                expr: [ WEIGHT_FACTOR=0.1, TARE_FACTOR=0.1 ]
              - mask: [ 1, 0, 0 ]
                expr: [ WEIGHT_FACTOR=0.01, TARE_FACTOR=0.01 ]
              - mask: [ 1, 0, 1 ]
                expr: [ WEIGHT_FACTOR=0.001, TARE_FACTOR=0.001 ]
              - mask: [ 1, 1, 0 ]
                expr: [ WEIGHT_FACTOR=0.0001, TARE_FACTOR=0.0001 ]
              - mask: [ 1, 1, 1 ]
                expr: [ WEIGHT_FACTOR=0.00001, TARE_FACTOR=0.00001 ]

              # bit offset 3, 2 bits
              - mask: [ 0, 1 ]
                expr: INCREMENT_SIZE=1
              - mask: [ 1, 0 ]
                expr: INCREMENT_SIZE=2
              - mask: [ 1, 1 ]
                expr: INCREMENT_SIZE=5

                expr: "IS_CHECKSUM_OK=IF(CHK2COMP7(0,17)=VALUE,1,0)"


                mask: [ 0, IS_HANDTARE, 1, @EW, IS_PRINTREQUEST, 3*@WF ]

              - mask: [ 0, 0, 0 ]

              - mask: [ 0, 0, 1 ]
              - mask: [ 0, 1, 0 ]
              - mask: [ 0, 1, 1 ]
              - mask: [ 1, 0, 0 ]
              - mask: [ 1, 0, 1 ]
              - mask: [ 1, 1, 0 ]
              - mask: [ 1, 1, 1 ]

              - mask: 0
                expr: [ WEIGHT_EXPANSION=1, TARE_EXPANSION=1 ]
              - mask: 1
                expr: [ WEIGHT_EXPANSION=10, TARE_EXPANSION=10 ]

                expr: WEIGHT=VALUE

                expr: TARE=VALUE

When the kbmMW Binary Parser is provided this definition file, it compiles it to build a parse tree, which efficiently can parse whatever you throw at it as a file or a stream.

We can see that one telegram mask has been defined in the TELEGRAMS/bytes array. It contains a mask that consists of 8 parts. Each part is, unless a * is included, exactly 1 byte wide.

The first part is 0x2 which simply means that the data must start with the hexadecimal value 2, which is STX (start of transmission) in the ASCII character set.

The second part is @SWA, which means that there must be one byte, which will be parsed by the tag called SWA.

The @SWB and @SWC also match one byte, that each of them must be parsed by a named tag.

Then we have the 6*@W part. That means that there are 6 bytes which must be parsed by the W tag.

You get the picture?

Let’s look at the SWA tag. It is defined as a bits tag. Hence it only parses bits and at most 8 of them. It has a mask defined as 0, 0, 1, 2*@IS, 3*@DP

That means that most significant bit should be 0, next one should also be 0, next should be 1, and then comes 2 bits which should be parsed by the IS tag, and then 3 lowest significant bits should be parsed by the DP tag.

Looking at the DP tag, you will see that is also a bits type tag, which makes sense since we are parsing a subset of bits from the SWA tag.

There are defined a number of possible DP bit masks, which, when matched, result in one or more expressions being executed.

So let’s say that the 3 bits matches 1 0 1. Then the expressions WEIGHT_FACTOR=0.001 and TARE_FACTOR=0.001 are both executed, essentially setting some values we can use later on, or explore from our program. Notice the []? In YAML that is called an inline array, where each element is separated by a comma. That is the reason why I mention that two expressions are executed in this case, when the match is successful.

The IS tag follow a similar procedure as the DP tag.

The SWB tag is an interesting one. It is used for parsing the 3rd byte of the data. It is also a bits type mask, and contains 8 parts, one for each bit in the matched byte.
The most significant bit should be 0. Whatever the next bit is, is set in the value IS_POWER_NOT_ZEROED, which can then be used in other expressions or by the developer later on. Then a 1 bit must be available.

Next comes a bit, which if set, sets IS_UNIT_UK_POUND to 1 and IS_UNIT_KILOGRAM to 0, else it sets IS_UNIT_KILOGRAM to 1 and IS_UNIT_UK_POUND to 0.

The next bit is set negated to the value IS_SETTLED. So if the bit was 1, then IS_SETTLED is set to 0 and visa versa.

The 3 remaining bits sets IS_OVERLOAD, IS_NEGATIVE and IS_NETTO values.

Simple stuff, right? 🙂

Now let us look at the W tag. It’s defined to be a string type tag, which means that any masks we write must be written as strings, and any value matched is seen as a string (a collection of bytes). As the W tag do not have any masks defined, the tag mask is considered a match, and any optional expression on that tag is run. In this case we just set the value WEIGHT equal to the complete matched value.

That introduce the magic word VALUE. It is a special variable, which always contains the latest match, regardless if it is a bits or strings match. In this case, it is how we get the

When all matches has been successful, we have a matching telegram, and only then will all the matching telegrams expressions be run. Internally kbmMW Binary Parser uses the kbmMemTable SQL and expression parser, and as such can do all the things that the expression parser can do, including calling functions etc.

We miss the code to run the parser.

        rd.OnSkipping:=procedure(var AByteCount:integer)
                          Log.Info('Skipping '+inttostr(AByteCount));

        // If you want to see the parsed values on a positive match.
        rd.OnMatch:=procedure(AValues:IkbmMWBPValues; var ABreak:boolean; const ASize:integer)
                         for i:=0 to High(a) do
                         if AValues.Value['IS_OVERLOAD'] then
//                         else if AValues.Value['IS_SETTLED'] then
//                              Log.Info('Unsettled gross:'+VarToStr(AValues.Value['GROSS']))
                             Log.Info('Gross:'+VarToStr(AValues.Value['GROSS'])+' Settled:'+vartostr(AValues.Value['IS_SETTLED']));
//                         ABreak:=true; // Only return first match.

        Log.Info('Found '+inttostr(rd.MatchCount)+' matches. Skipped '+inttostr(rd.SkippedBytes)+' bytes');

We take advantage of that a file reader is made available, that makes it easy to parse large files. But one could just as easily have created any other type readers, descending from TkbmMWBPCustomReader.

Each time the parser is not able to parse something successfully, it will attempt to skip past it, until either a match is made, or all data has been processed. The OnSkipping event is called on those occasions.

When a match is made, the OnMatch event is called. The developer can choose what to do with the values and if the parsing should continue when the event is done.

The file reader accepts one argument, the name of the definition file. And what is being read, is the file with the filename given in the Run statement.

Run will continue to run, until either all data has been read, or the process is interrupted, by either setting a zero value for AByteCount in OnSkipping, or setting ABreak to true in OnMatch.

After the execution ends, you can explore how many bytes was skipped and how many telegrams was read in total.

The parser is likely to evolve as new requirements appear, and I encourage users of it to play an active role in extending it so we all can benefit from a very versatile binary parser.