Digging for wORMs

kbmMW ORM and SQL rewriting explained in beautiful combination with kbmMW Smart services and kbmMW Smart clients.

I’ve promised to release v5 with a new very cool feature (in beta), which I’ll explain a bit about now: kbmMW ORM (object relational model).

Whats an ORM? It can be interpreted in multiple ways. In our way we interpret it as a way to persist, query and manipulate objects to and from a data storage of some sort.

We have focused more on the object persistence part of the ORM, than on the automatic relational management of master/detail scenarios. There are good and bad things about fully automatic relational cascade type storage and data manipulation. In some cases the bad things outweighs the good, and sometimes visa versa.

Show me the code!

Well.. its simple… create an object, tell the framework to persist the object and you have stored your data. It sounds too easy doesn’t it? It really _is_ easy.

This is an ordinary class, for a TPerson. This example contains some funny types, like kbmMWNullable, but thats just a convenience that makes some things even easier. However you could choose to use  plain simple regular native strings and integer and other native types.

type

 [kbmMW_Table('name:person, index:{name:i1,field:name,descending:false}, index:{name:i2,unique:true,fields:[{name:name,descending:true},{name:age}]')]
 TPerson = class
 public
  FID:kbmMWNullable;
  FName:kbmMWNullable;
  FAddress:kbmMWNullable;
  FAge:kbmMWNullable;

 published
  [kbmMW_Field('name:id, primary:true, generator:shortGuid',ftString,40)]
  property ID:kbmMWNullable read FID write FID;

  [kbmMW_Field('name:name',ftWideString,50)]
  property FullName:kbmMWNullable read FName write FName;

  [kbmMW_Field('name:address',ftWideString,50)]
  property Address:kbmMWNullable read FAddress write FAddress;

  [kbmMW_Field('name:age',ftInteger)]
  property Age:kbmMWNullable read FAge write FAge;
end;

initialization

 kbmMWRegisterKnownClasses([TPerson,TObjectList]);

end.

What you may also notice are some custom attributes named kbmMW_Table and kbmMW_Field.

kbmMW_Table tells the kbmMW framework something about how objects of this type are to be stored in data storage. It includes the desired table name and optional additional indexes that would be nice to have of performance reasons.

kbmMW_Field tells the framework that the following property or field is to be persisted, and under what data storage field name it will be persisted. Further it describes how that data storage field should be, type, size, precision and more.

The ID property has more information attached to it. The framework is being told that this field should be the primary field (a primary field should always be declared), and that the fields value is automatically generated upon storage in the database. In this case the value will contain a so called short GUID, which is a shortened version of a regular GUID.

The field refers to a generator. kbmMW ORM supports a number of generators:

  • GUID – Returns a GUID formatted as a regular GUID
    {123e4567-e89b-12d3-a456-426655440000}
  • SHORTGUID – Returns a short GUID where braces and dashes are missing:
    123e4567e89b12d3a456426655440000
  • SEQUENCE – Returns next unique number from a sequence. Provide name of sequencer in sequence property and optional sequencestart property (not supported by all databases!)
  • DATETIME – Returns a date time value, formatted according to the dateFormat property

In addition the field type can be set to ftAutoinc, just like any regular TDataset based databases. In this case the database will generate a unique number, which the framework will pickup automatically and return into the persisted object.

So how do we get around to actually persist anything?

First we create an ORM instance, linking it to a connection pool. The connection pool is responsible for the actual connection to the data storage.

 orm:TkbmMWORM;
...
 orm:=TkbmMWORM.Create(kbmMWSQLiteConnectionPool1);

In this case we have chosen a kbmMWSQLite connection pool, but most databases and connection pools are supported. I will write a bit more about that later.

The connection pool and a metadata component matching it (kbmMW will automatically attempt to find the right one if none has been setup) was put on the form at design time.

2017-04-17 18_47_59-SettingsOne single thing you need to setup on the connection pool, is the database name, and if you want the system to automatically create the SQLite database if none exists, then also set its SQLiteOptions property to contain mwsloAutoCreateDB, and set its MetaData to point on the kbmMWSQLiteMetaData1 component.

The metadata component is actually not mandatory to place on the form, and is only shown for completeness. kbmMW will automatically generate a metadata component instance that match the used connection pool, if none has been provided.

As you may be able to see from the screendump (if you have been using kbmMW before), is that this form is part of a simple n-tier server ORM sample which is included with v5.

Lets look at the code again… we now want the ORM to automatically create all relevant tables in the data storage. Im talking about table’s here, because that’s what most people use for storing data, but it could also be other types of storage, which have other concepts for storage than tables. As long as a connection pool, with accompanying metadata exists for that storage, it will work fine with kbmMW ORM.

Having the table created is as simple as this

orm.CreateTable(TPerson);

CreateTable can take an array of classes, if multiple tables should be created in one go.

You can test if a table (or tables) already exists using

 if not orm.ExistsTable(TPerson) then
....

CreateTable will only create tables if they are missing, but using ExistsTable may make it easier for you, to know when you can prepopulate newly created table with whatever bootstrap information you like to put into them from start.

You can use DeleteTable(TPerson) to delete the table and its data, and you can use PurgeTable to leave the table, but remove its contents.

How do we put some data into the table?

We do that by instantiating instances of TPerson like this

 p:=TPerson.Create;
 p.FullName:='Hans Hansen';
 p.Address:='Hans Home Address 1';
 p.Age:=65;
 orm.Persist(p);

After Persist has run successfully, you will find that p.ID has a value, even though we didn’t put one originally. If an object do not have a value for the field that was marked as primary, then that object is understood as a new object.

Thus in the future (for as long as we keep the instance pointed to by p) alive, kbmMW ORM will know that future persisting of it means that the existing record in the table will be updated rather than a new added.

To add more records, simply create more instances and persist them.

Remember that you own the TPerson instances that you create, and can do with them what you want, which also means that you are responsible for freeing them again when you no longer need them.

Freeing the instance will not affect what has been persisted.

If you make changes in the object and want those changes persisted, simply call orm.Persist(p) on the object again.

If you want to delete an object from persistence, you do

orm.Delete(p);

You still have the p instance in memory, but it is now gone from storage.

If you have a collection or an array of TPeople you can Persist and Delete those the same way as already shown.

Persisting data is no fun, unless you can retrieve them again at some point.

kbmMW ORM supports a vast number of ways to search and retrieve data via the various Query methods.

Lets say you want to retrieve an object with a specific primary key

var
 p:TPerson;
...
 p:=orm.Query<TPerson>('123e4567e89b12d3a456426655440000');

Now p will either be nil or point to a matching instance of TPerson. You own this object instance and may free it when you want.

Or if you know there is only 0 or 1 record in the table, you can simply do

p:=orm.Query<TPerson>;

Even if there are more records, only the first one will be returned to you. It should obviously be used with common sense because it does request all records from the data storage, and they may come in any order.

If you want a complete list of all persons known by the data storage

var
 p:TObjectList<TPerson>;
...
 p:=orm.QueryList<TPerson>;

Now… this is all nice and easy.. but what if I want to search for all people with a specific age?

var
 p:TObjectList<TPerson>;
...
 p:=orm.QueryList<TPerson>(['age'],[65]);

Now only people of the age 65 is returned.

var
 p:TObjectList<TPerson>;
...
 p:=orm.QueryList<TPerson>(['age'],[65],mwoqoGT);

Now only people older than 65 is returned. If you only wanted one of them you could have used Query<TPerson>… instead of QueryList.

You can specify multiple fields and values in your search this way, but all are controlled by the given operator (default mwoqoEQ = equal).

If you just wanted a count

var
 i:integer;
...
 i:=orm.QueryCount<TPerson>(['age'],[65],mwoqoGT);

You can use various aggregates, like QueryCount, QuerySum, QueryAvg, QueryMin, QueryMax and QueryStdDev or the generic QueryAggregate method.

Or if you really want the data as a TDataset instead, you can do

var
 ds:TDataset;
...
 ds:=orm.QueryDataset<TPerson>(['age'],[65],mwoqoGT);

This will return a dataset to, which you can use as any other dataset. The dataset is however not directly connected with the data storage behind, so changes in the dataset needs to be either resolved using kbmMW’s regular resolving mechanism (which is nice, but not ORM), or by converting the dataset back to a set of objects you can persist.

The field name given is the name your property is known as. So if you would have searched on a name, you would have specified ‘FullName’ as the field name, rather than ‘name’ as it’s given in the data storage.

With kbmMW ORM you must generally always refer to the object’s property names. There is an exception to that, which is shown further down.

For example if you want to get all deleted persons from the dataset, to let the ORM actually delete those:

var
 p:TObjectList<TPerson>;
...
 p:=orm.ListFromDataset<TPerson>(ds,[usDeleted]);
 orm.Delete(p);
 o.Free;

If you want to know if data has changed (been inserted or modified) in an object, you can

if orm.HasChanged(p) then
...

This is here where kbmMWNullable and TkbmMWDateTime comes into play. Regular native types like string, integer etc. can’t hold the value Null, and they also can’t hold any information about if their value has been changed.

But if you wrap the types in kbmMWNullable, or use TkbmMWDateTime instead of TDateTime, you suddenly have those abilities which the framework can take advantage of.

Looking at the FullName property in our TPerson object, you can see that it was wrapped in kbmMWNullable. That means we at any time can request information about if its null, or modified like this

if p.FullName.IsNull then
...
if p.FullName.Modified then
...

And we can clear its modified flag anytime if we need to

p.FullName.Modified:=False;

And we can set its value to null

p.FullName.Clear;

By default, all properties are marked as non modified when they have been returned as a result of one of the Query methods.

This way, Persist and other UpdateIfChanged / InsertIfChanged methods, will only resolve data back to the data storage if any changes have been made to the data in the object. If you are using regular non wrapped properties, then kbmMW ORM can only assume that the data may have changed, and it will attempt to update all provided objects regardless.

Most of the time, you really want to make more complex searches when querying for data. We have chosen to build in a uniform SQL based query language into kbmMW ORM which is not only easy to understand for most, but is automatically translated on the fly, to the query language used by the data storage.

var
 o:TObjectList<TPerson>;
...
 o:=orm.QueryList<TPerson>('SELECT * FROM uData.TPerson WHERE age>65 ORDER BY FullName'));

Notice that you select from uData.TPerson. That is a so called qualified class name, which means that there is no doubt about which exact TPerson you want to search on. You can have multiple TPerson classes defined in multiple units and that’s perfectly legal. While querying on them, just remember always to prefix the unit name.

Also notice the ORDER BY part. It refers to FullName which is the property in the TPerson class. The query parser also supports joins. But remember that only the fields that match the requested object type will be returned.

var
 o:TPersonAccount;
...
 o:=orm.Query<TPersonAccount>('SELECT p.id, name, value FROM uData.TPerson p, uData.TAccount a WHERE a.PID=p.ID');

To have all data returned, you will have to have define a TPersonAccount class somewhere which contains minimum the fields ID, Name and Value. Excess fields in the query will be silently ignored.

[kbmMW_VirtualTable]
 TPersonAccount = class
 private
   FID:string;
   FName:kbmMWNullable<string>;
   FValue:kbmMWNullable<double>;
 public
   [kbmMW_Field('name:id')]
   property ID:string read FID write FID;

  [kbmMW_Field('name:name')]
  property Name:kbmMWNullable<string> read FName write FName;

  [kbmMW_Field('name:value')]
  property Value:kbmMWNullable<double> read FValue write FValue;
 end;

Notice that this class has been given the attribute kbmMW_VirtualTable. A virtual table is a table that do not exist in the data storage, and as such if you try to CreateTable(TPersonAccount) an exception will be thrown telling you that its a virtual table that is only meant for transient uses.

Also notice that in this case we have defined ID as a regular native string instead of wrapping it with kbmMWNullable. As we will always have a value in the ID property, we can safely choose not to use kbmMWNullable, but its perfectly fine to wrap even this one with kbmMWNullable to follow a simple rule of thumb and not having to judge for each property type.

The kbmMW ORM SQL syntax is fairly complete, and sticking to it allow you to easily move your application between the different database types supported by kbmMW, in particular SQLite, Oracle, MSSQL, PostgreSQL, MySQL and Interbase/Firebird. Also many other databases are supported. If a database specific metadata component (which takes care of the rewriting of SQL) do not exist, then kbmMW ORM will attempt to fall back to a generic SQL metadata component, which will work in many cases.

However if there is a special query you want to use, which kbmMW ORM SQL do not directly support, you can send it raw, provided your orm instance will allow it.

To allow for custom native query statements, make sure that
orm.QueryMode:=mwoqmNative or mwoqmMixed.

It is default mwoqmMW.

Setting it to native, prevents kbmMW ORM to rewrite anything, and all query statements are considered native.

Setting it to mixed, will make kbmMW ORM assume that the query statements are kbmMW ORM SQL, but allow for you to put a # in front of a statement to send a native statement in a query.

var
 o:TObjectList<TPerson>;
...
 o:=orm.QueryList(TPerson,'#SELECT * FROM Person');

Notice that the table name given, must match the table name in the data storage.

But what about n-tier scenarios?

I started out with telling about that this article was based on an n-tier ORM sample, so it makes sense to show the remaining parts that enables a client to query for people, update changes to them and delete them.

kbmMW v5 was from outset designed to be the easiest to use n-tier product ever. With it came the new smart service and smart clients, and what else makes sense than to use those along with the ORM. This makes it possible to create both a server and a client, requiring only a few lines of business code almost totally eliminating boiler plate code (ie. code that does nothing for your business logic, but is required to glue things together).

Thus we will create a smart service with 3 methods, StorePersons, DeletePersons and GetPersons. As we have put the orm instance on the main form (Form1) we refer to that from here.

[kbmMW_Service('name:ORMDEMO')]
 TkbmMWCustomService2 = class(TkbmMWCustomSmartService)
 public
  // This lists stores changes (but not removed items) in the list of persons.
  [kbmMW_Method]
  procedure StorePersons(const APersons:TObjectList<TPerson>);

  // This lists deletes persons in the list.
  [kbmMW_Method]
  procedure DeletePersons(const APersons:TObjectList<TPerson>);

  // This method returns a list of persons.
  [kbmMW_Method]
  function GetPersons:TObjectList<TPerson>;
 end;

implementation

uses
 Unit1;

{$R *.dfm}

procedure TkbmMWCustomService2.StorePersons(const APersons:TObjectList<TPerson>);
begin
 // The APersons list is owned by this function, and must be manually freed.
 Form1.orm.Persist(APersons);
 APersons.Free;
end;

procedure TkbmMWCustomService2.DeletePersons(const APersons:TObjectList<TPerson>);
begin
 // The APersons list is owned by this function, and must be manually freed.
 Form1.orm.Delete(APersons);
 APersons.Free;
end;

function TkbmMWCustomService2.GetPersons:TObjectList<TPerson>;
begin
 // Return all persons (people).
 Result:=Form1.orm.QueryList<TPerson>;
end;

initialization
 TkbmMWRTTI.EnableRTTI(TkbmMWCustomService2);

This is really really simple isn’t it?  Only business logic code and nothing else.

In the FormCreate event handler of the main form Form1, where we have also put the orm creation, we add

 // Register all services automatically.
 // Services will only be autoregistered if they have a kbmMW_Service attribute.
 server.AutoRegisterServices;

In the event handler of the Listen button on Form1 we write

 server.Active:=not server.Active;

That’s all!

Now to the client.

2017-04-17 20_52_08-Project1 - RAD Studio 10.1 Berlin - Unit1

The client has a connect button, a button requesting a list of TPerson and a button to store changes made to TPerson data.

In addition it contains a string grid and a mtPersons kbmMemTable which are live bound together along with the navigator.

Embarcadero’s live binding is in some ways nice, but it is not terribly flexible, so it will only bind to native object properties and fields. In our case we have kbmMWNullable wrapped properties, which are simple to use in code, but which are not understood by livebinding.

This can be circumvented in a number of ways. One is to add a number of alternative properties to the TPerson object, which exposes the same data, but as native types.

This could look like this

[kbmMW_Table('name:person, index:{name:i1,field:name,descending:false}, index:{name:i2,unique:true,fields:[{name:name,descending:true},{name:age}]')]
 TPerson = class
 private
  FID:kbmMWNullable<string>;
  FName:kbmMWNullable<string>;
  FAddress:kbmMWNullable<string>;
  FAge:kbmMWNullable<integer>;

  function GetID:string;
  procedure SetID(AValue:string);
  function GetName:string;
  procedure SetName(AValue:string);
  function GetAddress:string;
  procedure SetAddress(AValue:string);
  function GetAge:integer;
  procedure SetAge(AValue:integer);
 published
  [kbmMW_Field('name:id, primary:true, generator:shortGuid',ftString,40)]
  property ID:kbmMWNullable<string> read FID write FID;

  [kbmMW_Field('name:name',ftWideString,50)]
  property FullName:kbmMWNullable<string> read FName write FName;

  [kbmMW_Field('name:address',ftWideString,50)]
  property Address:kbmMWNullable<string> read FAddress write FAddress;

  [kbmMW_Field('name:age',ftInteger)]
  property Age:kbmMWNullable<integer> read FAge write FAge;

  // One way of livebinding to kbmMWNullable or kbmMWDateTime fields is
  // to add an alternative access to them.
  // LiveBinding only supports basic data types.
  // Remember to kbmMW_Ignore the properties to tell
  // kbmMW not to attempt to stream them.
  // As kbmMWNullable and TkbmMWDateTime values retain
  // information about nullability and modification,
  // it may be preferred to use these types over
  // regular string, int, TDateTime etc. types.
  // If the number of properties of such types is large,
  // writing many alternative setters and getters may become
  // tedious. Instead you can (as this sample shows in unit1.pas), convert
  // the objects to a dataset (a proxy) which is easy to databind with.

  [kbmMW_Ignore]
  property LBID:string read GetID write SetID;

  [kbmMW_Ignore]
  property LBName:string read GetName write SetName;

  [kbmMW_Ignore]
  property LBAddress:string read GetAddress write SetAddress;

  [kbmMW_Ignore]
  property LBAge:integer read GetAge write SetAge;
 end;

Notice that the extra properties are not wrappe with kbmMWNullable. They have also been marked with the attribute kbmMW_Ignore, to tell the kbmMW framework that they are not to be marshalled (serialized).

Their only purpose is to allow for easy object live binding and is basically boiler plate code that we want to avoid.

There is another way that require less typing, and converting the objects to a dataset and back again when needed.

kbmMW ORM provides easy support for that.

In this case, we have already put a mtPerson kbmMemTable on the form at designtime, to make it easy to do designtime live binding.

All we need is to populate it with some data. This happens in the Get persons buttons eventhandler.

But first we need to prepare the smart client.

This can happen in the Connect buttons event handler.

// A form field
c:IkbmMWSmartClient;
...
// Connect button's event handler
c:=TkbmMWSmartRemoteClientFactory.GetClient(Transport,'ORMDEMO');

Then let us make a few methods to get and send data.

function TForm1.GetPersons:TObjectList<TPerson>;
begin
 // Request a list of persons.
 Result:=Use.AsObject<TObjectList<TPerson>>(c.Service.GetPersons);
end;

procedure TForm1.StorePersons(APersons:TObjectList<TPerson>);
begin
 // Store a list of persons.
 c.Service.StorePersons(Use.Arg(APersons));
end;

procedure TForm1.DeletePersons(APersons:TObjectList<TPerson>);
begin
 // Store a list of persons.
 c.Service.DeletePersons(Use.Arg(APersons));
end;

And in the Get Persons buttons event handler put

orm.ToDataset<TPerson>(mtPersons,GetPersons,true);

Now the grid is populated with data, and you can work the data in any way you want. As the data is available in a kbmMemTable, sorting, searching , filtering and other things are very easy to do.

At some point, the data may have been modified, and it should be sent to the server for updating.

In the Store Persons buttons event handler put

var
 persons:TObjectList<TPerson>;
begin
 persons:=orm.ListFromDataset<TPerson>(mtPersons,[usDeleted]);
 if persons.Count>0 then
   DeletePersons(persons)
 else
   persons.Free;
 persons:=orm.ListFromDataset<TPerson>(mtPersons);
 StorePersons(persons);
end;

This first sends a list of deleted TPerson’s to the server’s DeletePersons method (if any were deleted), and then a list of remaining data. In fact that list could also be limited to only inserted or modified data by adding the argument [usModified,usInserted] to the ListFromDataset method call. But right now we just ship all what is visible.

What happens on the server, is that it will know which fields was changed and only update the records that are relevant . Remember this require that all fields in the TPerson object are either wrapped with kbmMWNullable or are of type TkbmMWDateTime.

2017-04-17 21_16_09-Project1 - RAD Studio 10.1 Berlin - Unit1 [Running] [Built]

The combination of smart services, smart clients, advanced object marshalling, and the new ORM really makes it extremely easy to build multi tier applications in a true RAD way!

Happy wORMing!

We are not dead :)

But we are working hard on kbmMW v5 final release.

In addition to all the cool stuff unique to v5:

  • like the object notation framework and YAML, BSON, MessagePack support
  • like the smart services and smart clients
  • like the new much improved logging and logformatters
  • like the much improved object marshalling to and from XML, JSON, YAML, BSON and MessagePack
  • like the much improved XSD importer
  • like the hundreds of new features and fixes in existing code

we also want to add one new essential major brick to ensure that YOU can develop applications faster, cleaner, cooler and more intuitive with kbmMW than with any other similar tool!

We expect to release v5 Professional/Enterprise Edition final (with the new feature in beta), supporting Tokyo within the next week.

kbmMemTable Standard and Professional Edition supporting Tokyo will be released at the same time.

best regards
Kim/C4D

Android logcat tool

People developing for Android knows that checking it’s logfiles is a must, debugging any new application.

One can do it directly using adb.exe with the logcat command, but it outputs data to the console and is generally not really very useful due to the typical high amount of data produced by it.

Android Studio contains a log viewer application, but alas that requires installation of the complete Android Studio, which perhaps is overkill to check the contents of the log.

Some 3rdparty options also exists. Unfortunately they typically require Java or .Net to be available, and most of them are trials or limited commercials.

So when nothing exists that fulfills the requirements, the option is to make it yourself.

Thats what I did.

May I present kbmLogCat!

kbmLogCatIt is free and available for download from the C4D portal at https://portal.components4developers.com. if you do not have a login, please register and you will get immediate access to it.

kbmLogCat supports the various log buffers provided in Android, and contains advanced capture filter and live filter, storage and retrieval of log to/from a CSV file and sorting on the various columns.

It does not require Java or .Net to be installed.

When starting it first time, it will ask you for the path to the Android SDK.

If you have installed Delphi 10.1 Berlin with Android development option, you will typically find the SDK here:
C:\Users\Public\Documents\Embarcadero\Studio\18.0\CatalogRepository\AndroidSDK-2433_18.0.24468.8770

You can use any Android SDK that contains adb.exe in the platform-tools subdirectory.

If it stops logging, it may be because adb.exe has gotten confused. Simply stop and restart the kbmLogCat application.

Happy logging and debugging on Android!

 

DIRT CHEAP!!

As a tribute to Delphi’s 22 years birthday, and our own release of our Delphi and C++Builder middleware kbmMW v. 5.00.01 Beta, we have decided to go deep and dirty!

For a very limited time, until March 5 2017, you can get a new license of kbmMW Enterprise Edition, which is our flagship product with a whopping 75% discount!

What do you get with kbmMW Enterprise Edition?

The most complete n-tier product in existence for Delphi and C++Builder. Period!

  • Complete SOA native client support
  • Complete caching SOA server with loadbalancing, failover and proxying support.
  • Complete support for building your own application server.
  • Support for 36+ database API’s
  • Complete native and performant XML, JSON, YAML, BSON, Messagepack support.
  • Complete support for serializing and deserializing native Delphi objects to any of the above formats.
  • Native XSD importer which generates Delphi objects that can be serialized as described above.
  • Complete REST support with 5 lines of code. Spring boot inspired on steroids!
  • Native clients in PHP, Java, C#, K&R C, Apache Flex and more
  • Native extremely complete transactional queue based publish/subscribe messaging support with high performance memory and file based queues with auto healing functionality.
  • Native AMQP (eg. RabbitMQ and others) support
  • Native bundled support for numerous crypto technologies, like 3DES, Blowfish, Cast128, Cast256, DES, Ice, Ice2, IDEA, Mars, Misty1, RC2, RC4, RC5, RC6, Rijndael / AES, Serpent, ThinIce, Twofish
  • Native bundled support for numerous hashing technologies, like FNV1a, Haval, MD4, MD5, Murmur3, Ripe MD128, Ripe MD160, Ripe MD160, SHA-256, SHA-384, SHA-512, SHA1, Tiger, XX
  • Native bunled support for numerous compression technologies, like JPEG, LZH, LZ4, ZIP
  • Native high performance multi monitor Remote Desktop server and client code included, allowing creation of products like GotoMyPC, TeamViewer etc.
  • Complete authorization support based on resources, roles and actors.
  • Complete logging support.
  • Complete scheduler support for both high precision scheduling of very frequent tasks and scheduling of relaxed precision tasks on minutes, hours, day, monthly or yearly basis.
  • High performance dataset and data storage support
  • Support for automatic resolving of data changes from clients and other sources

Actually the bullets are of less importance… what is of importance is that you can build absolutely any product requiring a multi tier setup using kbmMW!

REST functionality? Takes 5 minutes to have a server running exposing even complex functionality.

Native clients? Takes 10 seconds to give them access to the same functionality as above, and you can add more specialized functionality as you need.

Cross platform clients? Again… 10 seconds then they are up and running!

Add more features while supporting older clients? Complete versioning of functionality is supported!

Need secure communication? Implemented in less than a minute, server and client side.

Need compression of data over slow lines? Again one minute.

You can do it all, and now you can do it DIRT CHEAP!

Visit http://www.components4developers.com to find the link to purchase DIRT CHEAP and to read more about kbmMW.

 

kbmMW v. 5.00.01 Beta released

We are happy to announce v5 of our popular middleware for Delphi and
C++Builder.

Yet again kbmMW continues to set the bar for what an n-tier product must be capable of in the real world!

We hope you will take the time to test this Beta release in your setup and report back regression errors to us by email at kbm@components4developers.com

Keywords for this release:

  • Huge number of new features and improvements!
  • New Smart services and clients for very easy publication of functionality and use from clients and REST aware systems without any boilerplate code.
  • New high quality random functions.
  • New high quality pronouncable password generators.
  • New support for YAML, BSON, Messagepack in addition to JSON and XML.
  • New Object Notation framework which JSON, YAML, BSON and Messagepack is directly based on, making very easy conversion between these formats and also XML which now also supports the object notation framework.
  • Lots of new object marshalling improvements, including support for marshalling native Delphi objects to and from YAML, BSON and Messagepack in addition to JSON and XML.
  • New LogFormatter support making it possible to customize actual logoutput format.
  • CORS support in REST/HTML services.
  • Focus on central performance improvements.
  • Pre XE2 compilers no longer officially supported.
  • Bug fixes

Look at end of post for detailed list of additions, changes and fixes.

Professional and Enterprise Edition is available for all with a current active SAU.
If your SAU has run out, please visit our shop to extend it with another 12 months.

CodeGear Edition is available for free, but only supports a specific Delphi/Win32 SKU, contains a limited feature set and do not include source.

Please visit https://portal.components4developers.com to download.

—-

kbmMW is the premiere n-tier product for Delphi, C++Builder and FPC on .Net, Win32, Win64, Linux, Java, PHP, Android, IOS, embedded devices, websites, mainframes and more.

Please visit http://www.components4developers.com for more information about kbmMW.

—-

Components4Developers is a company established in 1999 with the purpose of providing high quality development tools for developers and enterprises. The primary focus is on SOA, EAI and systems integration via our flagship product kbmMW.

kbmMW is a portable, highly scalable, high end application server and enterprise architecture integration (EAI) development framework for Win32, ..Net and Linux with clients residing on Win32, .Net, Linux, Unix, Mainframes, Minis, Embedded and many other places. It is currently used as the backbone in hundreds of central systems, in
hospitals, courts, private, industries, offshore industry, finance, telecom, governements, schools, laboratories, rentals, culture institutions, FDA approved medical devices, military and more.

5.00.00 Beta Feb 19 2017

Important notes (changes that may break existing code)
======================================================
* Officially now only supporting XE2 and forward. Support for pre XE2 may not be complete.

New stuff
=========
– Officially now only supporting XE2 and forward. Support for
pre XE2 may not be complete.
– Added RemoteLocationsPrimaryPart to
TkbmMWAuthorizationTransportConstraint (default false).
If set to true, will only compare primary part of
clientidentity.remotelocation.
Primary part is defined by everything up til the last ‘:’. In most
cases that will mean the IP address without the port number.
– Added IkbmMWLogFormatter interface to kbmMWLog.
– Added TkbmMWStandardLogFormatter to kbmMWLog. It contains vastly
improved features for controlling the look and layout of the log
output. Its the default used log formatter.
– Added TkbmMWSimpleLogFormatter to kbmMWLog. It outputs a log with
only datetime, type (info/warning/error/fatal etc) and log string.
– Added LogFormatter property to IkbmMWLogManager. It can be set to a
custom log formatter.
– Added mwldtRaw log data type and methods LogRaw to IkbmMWLog. It
allows for logging bytes or strings without
interpretation/reformatting.
– Added global SystemLog instance which will always output to system
log outputs. It is used as a fallback in case the regular log system
excepts.
– Added support for marshalling/unmarshalling named enum values via
the new kbmMW_Enum attribute.
Its useful when an enum value cant be used as an identifier or its
value should be different from its presented name.
– Added ExceptOnUnknownType to TkbmMWCustomRTTIMarshal. It controls if
to throw an exception if an unknown type is accessed or not
(default true).
– Added support for marshalling/unmarshalling TDictionary<K,V> types.
– Added support for correctly instantiating classes in unmarshalling
that contains a capacity constructor argument.
– Added Run methods to TkbmMWScheduler. It allows for one time run and
forget scheduled async code. They are used in the same way as the
Schedule methods, but sets Occurs to mwsoRun.
The job will be automatically unscheduled after the run.
– Added overloaded DelayInitial(const AInitialDelaySecs:double) to
IkbmMWScheduledEvent.
It allows for directly giving an initial delay in seconds. It
supports fractional second values.
– Added Clear to TkbmMWScheduledEvents (scheduler.Events) to allow for
clearing all scheduled events. Running events will be allowed to
finish.
– Added GetNameSpaceByURI, DefaultNameSpace, DefaultNameSpaceURI to
TkbmMWDOMXMLNameSpaceList.
– Added SearchRelNodeByID, SearchNodeByID, SearchRelNodeByGivenName,
SearchNodeByGivenName to IkbmMWDOMXMLParser.
– Added ExpandNameSpace, GivenName, ChildrenByGivenName,
ChildByGivenName, CheckChildByGivenName to TkbmMWDOMXMLNode.
– Added CheckNodeByGivenName, NodeByGivenName to TkbmMWXMLNodeList.
– Added missing ADSDBF support to kbmMWReg.
– Added mwrtoOptionalInsert,mwrtoKeyNotNullInsert,
mwrtoOptionalModify,mwrtoKeyNotNullModify,
mwrtoOptionalDelete,mwrtoKeyNotNullDelete and table modifier flags
OIM/OMI (optional insert/modify), OID/ODI (optional insert/delete),
OMD/ODM (optional modify/delete), OI (optional insert),
OD (optional delete), OM (optional modify),
KIM/KMI (insert/modify on key not null), KID/KDI (insert/delete on
key not null),
KMD/KDM (modify(delete on key not null), KI (insert on key not null),
KM (modify on key not null), KD (delete on key no null)
to TkbmMWResolverTableOption
They allow for more finegrained definition on when insert/update
/delete should happen.
If one of the optional (OIM,OMD,OID,OI,OD,OM) is given, no exception
will be raised for the table when an operation did not succeed, and
resolving will continue. If one of the key (KIM,KMD,KID,KI,KM,KD) is
given, record resolve operation will be skipped for the specific
record if key is null.
– Added ReadBOM overloaded methods with default TEncoding option to
TkbmMWPlatformMarshal.
– Added overloaded kbmMWTextToString, kbmMWExpectText,
kbmMWStringRightPad, kbmMWStringLeftPad,
kbmMWGetComputerName functions to kbmMWGlobal.pas.
– Added optional ACallback anonymous function to SendAsyncRequest and
SendAsyncRequestEx in kbmMWClient.pas (WIB only).
It allows for providing an anonymous function that should be called
when an async response is returned. The function must follow this
type:
function(Sender:TObject;
TransportStream:IkbmMWCustomTransportStream):boolean;
– Added LOB blob/clob support (user fix) to DOA adapter.
– Vastly improved XSD importer.
– Added generic Object Notation framework for building virtual object
trees consisting of TkbmMWONNative, TkbmMWONArray and TkbmMWONObject
which all are based on TkbmMWONCustomObject.
– Added YAML parser/generator based on object notation framework.
– Added Messagepack parser/generator based on object notation
framework.
– Added BSON parser/generator based on object notation framework.
– Added support for converting XML to and from object notation format
via LoadFromObjectNotation and SaveToObjectNotation functions.
– Added new TkbmMWCustomSmartService and TkbmMWCustomHTTPSmartService
services, which can be selected in the service wizard to create
smart services. Smart services supports tagging any function to be
exported to a smart client or a REST client
(TkbmMWCustomHTTPSmartService required).
– Updated transports to support marshalling objects via the transport.
– Added several near lock free features to kbmMWGlobal, including
TkbmMWLockFreeStack, more interlocked functions, lock free array GC
fixes, optional TkbmMWREWLock statistics, support for TkbmMWMREWLock
can switch to TMonitor for reader/writer fair support.
– Optimized many algorithms like TkbmMWStringBuilder,
TkbmMWGenerateGUID, TkbmMWGenerateShortGUID and many more to improve
performance overall.
– Added many new general purpose functions to kbmMWGlobal.
– Added direct support for salt in hash functions deriving from
TkbmMWCustomHash.
– Added support for anonymous functions in async kbmMWClient requests
(WIB)
– Added new kbmMWRTTI unit with loads of RTTI functionality.
– Added YAML stream format for kbmMemTable and descendants.
– Added Messagepack stream format for kbmMemTable and descendants.
– Added BSON stream format for kbmMemTable and descendants.
– Added object marshalling to and from YAML.
– Added object marshalling to and from Messagepack.
– Added object marshalling to and from BSON.
– Added ServicePath property to TkbmMWClientIdentity.
– Added MaxLogins to TkbmMWAuthorizationManager which is default 100
to prevent potential login spam.
– Added support for HTTPSys based transport for Windows. Its specially
useful for high performance RESTful applications.
– Added AutoRegisterServices method to TkbmMWServer which can be used
for automatically locating and registering any service that has the
kbmMW_Service attribute defined. It makes use of the old
RegisterService and RegisterServiceByName optional.
– Added LocalSinceEpochMS, UTCSinceEpochMS, Temporenc to
TkbmMWDateTime. Temporenc is a very compact binary encoding of a
datetime value.
– Added FastSetUTC method to TkbmMWDateTime to threadsafely very fast
setting the UTC value of an already defined TkbmMWDateTime record.
The record MUST be initialized before using FastSetUTC, for example
by dt:=TkbmMWDateTime.Now;
– Added ValueToStream, ValueFromStream, ValueToBytes and
ValueFromBytes in
TkbmMWCustomRTTIMarshal. Allows for easy marshalling objects to and
from bytes and streams.
– Added ExceptOnUnknownType boolean property to
TkbmMWCustomRTTIMarshal.
Controls if an exception should be thrown in case a type cant be
marshalled or unmarshalled, or it should be silently skipped.
– Added Devart MyDAC support (DMYDAC).
– Added CORS support (Cross Origin-Resource Sharing) in
TkbmMWCustomHTTPService and descendants. (OnCORS event).
– Added PerformOPTIONS and PerformPATCH support to
TkbmMWCestomHTTPService.
(OnOptions and OnPatch in TkbmMWEventHTTPService).
– Added REST transport stream format which must be used if non HTTPSys
transport is used for smart services.
– Added TkbmMWSmartClient which allows simple and easy access to
server side smart functions.
Check simpleinvocation sample.
– Added high quality random functions in new kbmMWRandom unit.
TkbmMWRandomDelphi (32 bit non high quality ramdom!),
TkbmMWRandomSplitMix (64 bit)
TkbmMWRandomXoroshiro128Plus (64 bit),
TkbmMWRandomXoroshiro1024 (64 bit),
TkbmMWRandomPCGUInt32 (32 bit),
TkbmMWRandomMersenneTwisterUInt32 (32 bit),
TkbmMWRandomMersenneTwisterUInt64 (64 bit)
– Added high quality pronouncable password generators in new unit
kbmMWPassword.
TkbmMWMixerPasswordGen, TkbmMWKoremutakePasswordGen
– Added support for ExchangeType in TkbmMWAMQPClient.
– Added new GC and CloseAndGC methods in TkbmMWAMQPChannel to force
getting rid of defunct/cached but unused AMQP channels.
– Added IPVersion property to TkbmMWCustomTCPIPIndyClientTransport.

Changes/minor additions
=======================
– Removed (mwloProcessThreadInfo,mwloThreadName from TkbmMWLogOption
since its now controlled by the log formatter.
– Changed Activate method of IkbmMWScheduledEvent to have true as
default argument AValue.
– Changed SetAsDuration and GetAsDuration to use TkbmMWDuration
instead of TkbmMWDateTime.
– Added support for checking if path is accessible in
TkbmMWBufferedFileStream.Create.
– Changed JSON parser/generator to be based on the object notation
framework.
– Improved scalability of FastCGI services to better spread load
accross multiple instances.

Fixes
=====
– Fixed A/V when attempting to use audit file.
– Fixed missing clearing of TkbmMWDuration when parsing duration
strings, resulting in an incorrect duration value (instead of null)
on invalid duration strings.
– Fixed TDuration.Passed(const ADuration:TkbmMWDuration) which didnt
return correct value.
– Fixed deactivating precise scheduled events.
– Fixed events scheduled to start on day, month, week or year
boundaries.
– Fixed issues to initial delay in scheduled events.
– Fixed parsing tags crossing line boundaries in XML parser.
– Fixed various namespace issues in XML parser.
– Fixed WIB compilation issue for XE.
– Fixed TkbmMWStreamStore offset bug when writing first segment.
– Fixed potential shutdown issue in Indy10 transports due to Indy10
bug.
– Fixed kbmMWDependency non accumulating memory leaks.
– Fixed NexusDB transport compilation.
– Fixed Autoinc field detection in EDB (ElevateDB) adapter.

Teaser: kbmMW safety first… Random numbers and great passwords!

Usually Delphi’s built in 32 bit random generator is sufficient for most tasks, like generating random numbers for some tests, or a game or something similar. However the random generator is, if one focus on security, not strong enough to be used for cryptographic uses, like password generation.

Random numbers

In next version of kbmMW, a set of random number generators has been added for generating 32bit and 64bit random values in addition to supporting the standard Delphi Randon method.

32 bit random generators in kbmMW:

  • TkbmMWRandomDelphi
  • TkbmMWRandomPCGUInt32
  • TkbmMWRandomMersenneTwisterUInt32

64 bit random generators in kbmMW:

  • TkbmMWRandomSplitMix
  • TkbmMWRandomXoroshiro128Plus
  • TkbmMWRandomXoroshiro1024
  • TkbmMWRandomMersenneTwisterUInt64

They all follow the same principle, and it’s easy to replace one with another, or even add another custom random generator.

To see the randomness of the generators, one can make a lot of random X and Y values and plot them in a fine masked grid. A perfectly random function (in reality that does not exist in current computers) ought to spread the numbers fairly evenly across the grid.

2017-02-01-01_46_40-xbox
The 32 bit standard Delphi random plot

2017-02-01-01_48_03-xbox
The 32 bit PCG random plot

2017-02-01-01_49_17-xbox
The 32 bit Mersenne Twister random plot

2017-02-01-01_50_42-xbox
The 64 bit split mix random plot

2017-02-01-01_52_53-xbox
The 64 bit Xoroshiro 128+ random plot

2017-02-01-02_14_37-xbox
The 64 bit Xoroshiro1024 random plot

2017-02-01-02_17_03-xbox
The 64 bit Mersenne Twister random plot

As can be seen, the Delphi random generator is pretty bad in producing values across the entire value space, while the others show a much more even spread as would be expected by a fair random generator.

Choosing a random number generator for a particular purpose can be difficult. One has to weigh in speed vs randomness vs full cycle time and other factors. Basically the perfect randon number generator do not exist at this time for regular computers.

These pages explains a bit about their weaknesses and strengths:
http://xoroshiro.di.unimi.it/
http://www.pcg-random.org/

But at least now you have a better choice. And in fact this brings us to the next section. Generation of strong passwords.

Strong passwords

Most software today supports some sort of user login, where a password is required. kbmMW certainly supports that, through for example the authorization manager.

But the challenge is to force users to use some good passwords, which they still have a chance to remember without writing down.

As a first for Delphi, kbmMW now also supports multiple password generator algorithms to encourage use of strong passwords.

kbmMW comes with two password generators and a framework on which new custom password generators can be built:

  • TkbmMWMixerPasswordGen
  • TkbmMWKoremutakePasswordGen

The Mixer password generator supports selecting the minimum and maximum number of digits, minimum and maximum number of punctuation characters, unicase or mixed case, and minimum length of a generated password.

The Koremutake password generator generates a random 64 bit value, and converts that 64 bit value into a string consisting of two and three character character groups, forming a semi pronounceable password.

Using the password generators is very simple:

var
   pg:TkbmMWMixerPasswordGen;
begin
     pg:=TkbmMWMixerPasswordGen.Create;
     try
        // Optionally set length, digits, punctuation and case settings.
        // Default a password will be minimum 8 characters long,
        // contain from 1 to 4 digits, no punctuation characters
        // and use mixed case alpha characters.
        Memo1.Lines.Add(pg.Generate);
     finally
        pg.Free;
     end;
end;

This could output: 69vcRPhw

var
   pg:TkbmMWKoremutakePasswordGen;
begin
     pg:=TkbmMWKoremutakePasswordGen.Create;
     try
        Memo1.Lines.Add(pg.Generate);
     finally
        pg.Free;
     end;
end;

This could output: GUMIPAVYGRYTIFOFYSI

We will continue to monitor the various options for password generators, and provide support for them as we find them interesting.

Teaser : kbmMW’s new object notation

kbmMW now supports XML, JSON, BSON, Messagepack and YAML natively, and includes a new common Object Notation format for easy conversion.

JSON and XML has been supported natively by kbmMW for years. Later kbmMW also supported marshalling of native Delphi objects to and from JSON and XML.

In addition to it being a very nice way to stream Delphi objects, the feature also allowed for a way to convert XML to JSON and visa versa.

However, the upcoming version of kbmMW now includes a new object notation framework, that makes it easy to convert from one notation to another, without going through a native Delphi object.

The new version of kbmMW not only supports JSON and XML, but now also YAML (which is a more human readable format) and the compact binary Messagepack and BSON formats, natively, both for creating your own object notation trees in code, and for marshalling Delphi objects to and from.

With the many new object notation formats supported by kbmMW, it also made sense to find a way to make it easy to convert between them, without having to marshal to and from native Delphi objects.

For that reason the kbmMW Object Notation format was invented. The object notation format is based on a number of classes:

  • TkbmMWONCustomObject (from which the classes listed below descends)
  • TkbmMWONObject (for key/value type data)
  • TkbmMWONArray (for array type data)
  • TkbmMWONNative (for string, integer, floats, date/time, binary values, custom native values and more)

In addition a number of abstract stream classes was developed, which purpose is to handle converting the object notation trees to and from various string or binary formats, and a set of abstract classes that handles marshalling object notation trees to and from native Delphi objects.

JSON, YAML, BSON and Messagepack object notation classes directly descends from the Object Notation classes.

For JSON:

  • TkbmMWJSONObject
  • TkbmMWJSONArray
  • TkbmMWJSONNative
For YAML:

  • TkbmMWYAMLObject
  • TkbmMWYAMLArray
  • TkbmMWYAMLNative
For BSON:

  • TkbmMWBSONObject
  • TkbmMWBSONArray
  • TkbmMWBSONNative
For Messagepack:

  • TkbmMWMessagepackObject
  • TkbmMWMessagepackArray
  • TkbmMWMessagepackNative

Since XML is a more rich notation format, it was decided to leave its SAX based parser (which the XML DOM class use) and instead add support for converting the XML DOM tree to and from the Object Notation format. This way its easy to convert between different notation formats, without sacrificing the more advanced features of our native XML parser.

The above specialized object notation classes, are in reality very very slim.
Thus the underlying TkbmMWON…. classes are sufficient for conversion. So even if one build a JSON object notation tree in code, it can still be understood by the Messagepack formatter code etc.

Generally it’s recommended using the non specialized classes instead of the specialized ones for simplicity reasons.

So lets try to build a simple object notation tree:

var
 father,child:TkbmMWONObject;
 children:TkbmMWONArray;
begin
 father:=TkbmMWONObject.Create;
 father.AsString['name']:='Joe Simpson';
 father.AsInt32['age']:=42;
 children:=TkbmMWONArray.Create;
 child:=TkbmMWONObject.Create;
 child.AsString['name']:='Joe Simpson Jr';
 child.AsInt32['age']:=12;
 children.Add(child);
 father.AsArray['children']:=children;
 ...

This basically makes a father object with name and age, and a list of children (containing only one child with name and age). The object notation tree should now look like this:

2017-02-01-01_09_51-xbox

So lets convert this to JSON:

var
 json:TkbmMWJSONStreamer;
 s:string;
begin
 json:=TkbmMWJSONStreamer.Create;
 try
  // If set to true, an exception will be raised in case a type is not supported
  // by the specific streamformat. An event also exists which one can hook
  // into, to get information about what type fails etc.
  json.ExceptOnUnknownType:=true;
  s:=json.SaveToUTF16String(father);

  // s now contains the serialized object in JSON format.
 finally
  json.Free;
 end;
end;

Resulting in s containing:

{"name":"Joe Simpson","age":42,"children":[{"name":"Joe Simpson Jr","age":12}]}

or to YAML:

var
 yaml:TkbmMWYAMLStreamer;
 s:string;
begin
 yaml:=TkbmMWYAMLStreamer.Create;
 try
  // If set to true, an exception will be raised in case a type is not supported
  // by the specific streamformat. An event also exists which one can hook
  // into, to get information about what type fails etc.
  yaml.ExceptOnUnknownType:=true;
  s:=yaml.SaveToUTF16String(father); 

  // s now contains the serialized object in YAML format.
 finally
  yaml.Free;
 end;
end;

Resulting in s containing (with linefeeds):

name: Joe Simpson
age: 42
children:
  -
    name: Joe Simpson Jr
    age: 12

BSON and Messagepack follows the same pattern as JSON and YAML.

And for XML:

var
 xml:TkbmMWDOMXML;
 s:string;
begin
 xml:=TkbmMWDOMXML.Create;
 try
  xml.LoadFromObjectNotation(father);
  xml.Top.Name:='father';
  xml.AutoIndent:=true;
  xml.AutoLineFeed:=true;
  xml.Typed:=false;
  s:=xml.SaveToString;

  // s now contains the serialized object in XML format.
 finally
  xml.Free;
 end;
end;

Resulting in s containing (with linefeeds):

<father>
 <name>Joe Simpson</name>
 <age>42</age>
 <children i="0">
  <name>Joe Simpson Jr</name>
  <age>12</age>
 </children>
</father>

In BSON, the output would be a binary value (a stream or a byte array). A Base64 encoded version of the above object notation tree would be:

XQAAAAJuYW1lAAwAAABKb2UgU2ltcHNvbgAQYWdlACoAAAAEY2hpbGRyZW4ALwAAAAMwACcAAAACbmFtZQAPAAAASm9lIFNpbXBzb24gSnIAEGFnZQAMAAAAAAAA

and a Base64 encoded Messagepack stream would look like:

g6RuYW1lq0pvZSBTaW1wc29uo2FnZSqoY2hpbGRyZW6RgqRuYW1lrkpvZSBTaW1wc29uIEpyo2FnZQw=

The Object Notation classes also supports letting the developer indicate if a property key/value pair of type TkbmMWONNative should be streamed as an attribute or an element in XML.

Further the TkbmMWONNative class supports custom values, like the BSON Decimal128 value. As no other streamers understand that value type, using custom types do limit lossless conversion between different object notation formats.

Delphi native objects can be marshalled to and from the Object Notation format, and thus also to and from the above supported specialized object notation formats formats.