What is Hazelcast Auto DB Integration?

Hazelcast Auto DB Integration is a development tool for projects involving connecting existing RDBMS to Hazelcast IMDG.

It can be used to automatically extract metadata from an existing database and generate code that supports features of the underlying database, the Hazelcast client, and IMDG. It provides an automatically generated domain model including POJOs (Portable), SerializationFactories, ClassDefinitions, MapStores, MapLoaders, ClientConfiguration, Ingest, Index and more. The generated domain model is compatible with Hazelcast Enterprise functions such as High-Density Memory Store, Hot Restart Store, Management Center etc.

Hazelcast Auto DB Integration is a part of the Speedment Enterprise Suite since version 3.1.13.

Main Features

Using the Hazelcast Auto DB Integration, Speedment can greatly simplify working with Hazelcast and can, from an existing database, automatically generate:

  • Java domain data model (e.g. entities)
  • Hazelcast Serialization support
  • Hazelcast MapStore/MapLoad support
  • Hazelcast configuration handling
  • Hazelcast indexing based on the underlying database indexing
  • Persistence handling

In addition to this, the Hazelcast Auto DB Integration also provides:

  • Automatic ingest of data from an existing database to the Hazelcast grid
  • Access to the Hazelcast grid for additional languages such as
    • C++
    • node.js
    • C#
    • Go
    • Python
    • and many other languages

Architecture

The Hazelcast Auto DB Integration support storage of entities in distributed maps in a client/server architecture whereby the HazelcastBundle runtime only needs to reside on the client side. No extra software is required on the server side which allows easy setup, migration and management of Hazelcast clusters.

Hazelcast Architecture
The Hazelcast client/server architecture

Starting from the bottom of the picture, data in this example is stored in a traditional database in two tables named “film” and “actor”. Using the HazelcastBundle data from these tables can, via the application, easily be ingested into the Hazelcast server grid. in this example, the grid consist of two nodes “Hazelcast Server Node #0” and “Hazelcast Server Node #1”. Node #0 and #1 each holds approximately 50% of the data from the database tables in two different distributed maps. The application can query and manipulate data in the Hazelcast server grid without touching the database.

Since the database is no longer involved in querying, application speed may be greatly improved in many cases.

Consistency across Domains

The database is considered the “Source of Truth” and the Hazelcast server grid can reflect a consistent view of the underlying database at an unspecified time. Thus, it is wrong to use data from one domain and then use it in the other domain.

If updates are made to the database by a non-Auto DN Integration application, those updates are not reflected in the Hazelcast server grid and a new ingest of data must be done to ensure consistency.

Obtaining a license

In order to use Hazelcast Auto DB Integration you need a commercial license or a trial license key. The most straight forward way of obtaining a trial license is to download a pre-initialized project via the Hazelcast Auto DB Integration Initializer.

From that page, a minimalistic starter project can be downloaded. The included pom.xml file can be used as it is or be merged with an existing project. The first time the code generation tool is started, the user is prompted to apply for a trial license.

For more details, see the general information about how to handle licenses in Speedment products.

Installing the Hazelcast Bundles

There are two Hazelcast bundles that need to be installed:

  • HazelcastToolBundle that is needed by the UI Tool to generate entity classes and other support classes (generation)
  • HazelcastBundle that is needed at runtime by the Hazelcast client application (runtime)

Installing the HazelcastToolBundle

In the pom.xml file, the speedment-enterprise-maven-plugin configuration needs to be updated so that the HazelcastToolBundle class is added and the hazelcast-tool dependency is added:

<plugins>
    <plugin>
        <groupId>com.speedment.enterprise</groupId>
        <artifactId>speedment-enterprise-maven-plugin</artifactId>
        <version>${speedment.version}</version>

        <configuration>
            <components>
                <!-- Add the following component to this plugin -->
                <component>com.speedment.enterprise.hazelcast.tool.HazelcastToolBundle</component>
            </components>
            <appName>${project.artifactId}</appName>
            <packageName>${project.groupId}</packageName>
        </configuration>

        <dependencies>
            <dependency>
                <groupId>mysql</groupId>
                <artifactId>mysql-connector-java</artifactId>
                <version>${mysql.version}</version>
                <scope>runtime</scope>
            </dependency>
            
            <!-- The dependency below needs to be added -->            
            <dependency>
                <groupId>com.speedment.enterprise.hazelcast</groupId>
                <artifactId>hazelcast-tool</artifactId>
                <version>${speedment.version}</version>
            </dependency>

        </dependencies>
    </plugin>
</plugins>

Installing the HazelcastBundle

In the pom.xml file, the following dependencies needs to be added to make the HazelcastBundle present on the classpath:

<dependencies>

    <!-- other dependencies -->

    <dependency>
        <groupId>com.speedment.enterprise.hazelcast</groupId>
        <artifactId>hazelcast-runtime</artifactId>
        <version>${speedment.version}</version>
    </dependency>

    <dependency>
        <groupId>com.hazelcast</groupId>
        <artifactId>hazelcast-client</artifactId>
        <version>3.11</version>
    </dependency>
</dependencies>

In the application builder, the HazelcastBundle needs to be added to allow injection of the Hazelcast runtime components as shown in this example:

final Speedment hazelcastApp = new SakilaApplicationBuilder()
    .withPassword("sakila-password")
    .withBundle(HazelcastBundle.class)
    .build();

Entities

Hazelcast compatible data entities are automatically generated from the database metadata. The generated entities implements Hazelcast’s Portable interface. Hazelcast serialization and Map handling involves several aspects as described in this chapter. In all the code examples below, the Sakila sample database are being used. The Sakila database is also available as a Docker instance.

Serialization

The HazelcastToolBundle automatically generates the necessary entity POJO objects including Portable serialization. For example, the implementation of the Film interface is generated like this:

public final class FilmImpl extends GeneratedFilmImpl implements Film {}

As can be seen, an entity implementation class just inherits all its methods from another generated class. This allows the possibility to override generated methods with custom code that is retained between re-generation of code.

@GeneratedCode("Speedment")
public abstract class GeneratedFilmImpl implements Film {
    
    private int filmId;
    private String title;
    private String description;
    private Date releaseYear;
    private short languageId;
    private Short originalLanguageId;
    private short rentalDuration;
    private BigDecimal rentalRate;
    private Integer length;
    private BigDecimal replacementCost;
    private String rating;
    private String specialFeatures;
    private Timestamp lastUpdate;
    
    protected GeneratedFilmImpl() {}
    
    @Override 
    public int getFilmId() { return filmId; }
    
    @Override 
    public String getTitle() { return title; }
    
    @Override 
    public Optional<String> getDescription() { return Optional.ofNullable(description); }
    
    // Rest of getters hidden for brevity
    
    @Override
    public Film setFilmId(int filmId) {
        this.filmId = filmId;
        return this;
    }
    
    @Override
    public Film setTitle(String title) {
        this.title = title;
        return this;
    }
    
    @Override
    public Film setDescription(String description) {
        this.description = description;
        return this;
    }    
    // Rest of setters and finders hidden for brevity
    
    @Override 
    public int getFactoryId() { return 1321754994;}
    
    @Override 
    public int getClassId() { return 3143044; }
    
    @Override
    public void writePortable(PortableWriter writer) throws IOException {
        writer.writeInt("film_id", getFilmId());
        writer.writeUTF("title", getTitle());
        writer.writeUTF("description", getDescription().orElse(null));
        if (getReleaseYear().isPresent()){
            writer.writeBoolean("__null__release_year", false);
            writer.writeLong("release_year", getReleaseYear().get().getTime());
        } else {
            writer.writeBoolean("__null__release_year", true);
            writer.writeLong("release_year", Long.MIN_VALUE);
        }
        writer.writeShort("language_id", getLanguageId());
        if (getOriginalLanguageId().isPresent()){
            writer.writeBoolean("__null__original_language_id", false);
            writer.writeShort("original_language_id", getOriginalLanguageId().get());
        } else {
            writer.writeBoolean("__null__original_language_id", true);
            writer.writeShort("original_language_id", Short.MIN_VALUE);
        }
        writer.writeShort("rental_duration", getRentalDuration());
        writer.writeUTF("rental_rate", getRentalRate().toString());
        if (getLength().isPresent()){
            writer.writeBoolean("__null__length", false);
            writer.writeInt("length", getLength().getAsInt());
        } else {
            writer.writeBoolean("__null__length", true);
            writer.writeInt("length", Integer.MIN_VALUE);
        }
        writer.writeUTF("replacement_cost", getReplacementCost().toString());
        writer.writeUTF("rating", getRating().orElse(null));
        writer.writeUTF("special_features", getSpecialFeatures().orElse(null));
        writer.writeLong("last_update", getLastUpdate().getTime());
    }
    
    @Override
    public void readPortable(PortableReader reader) throws IOException {
        setFilmId(reader.readInt("film_id"));
        setTitle(reader.readUTF("title"));
        setDescription(reader.readUTF("description"));
        setReleaseYear(reader.readBoolean("__null__release_year")
            ? null
            : new Date(reader.readLong("release_year")));
        setLanguageId(reader.readShort("language_id"));
        setOriginalLanguageId(reader.readBoolean("__null__original_language_id")
            ? null
            : reader.readShort("original_language_id"));
        setRentalDuration(reader.readShort("rental_duration"));
        setRentalRate(new BigDecimal(reader.readUTF("rental_rate")));
        setLength(reader.readBoolean("__null__length")
            ? null
            : reader.readInt("length"));
        setReplacementCost(new BigDecimal(reader.readUTF("replacement_cost")));
        setRating(reader.readUTF("rating"));
        setSpecialFeatures(reader.readUTF("special_features"));
        setLastUpdate(new Timestamp(reader.readLong("last_update")));
    }
    
    // toString, equals and hashCode hidden for brevity
}

Because Portable objects cannot handle nullable wrapper classes in Hazelcast, these fields are handled with an extra synthetic fields beginning with __null__ (two understrokes at both ends).

Primary Keys

Because entities are stored in distributed maps, each entry must have a unique key. The key is extracted from an entity depending on its primary key(s) (if any). This works in the following way:

No Primary Key

A synthetic key is used in the form of a Long that generates a unique value each time an entry is added to the map. The specific sequence is undefined but it is guaranteed keys are never repeated for the same Map within the same cluster. It is not recommended to use entities with no primary key.

One Primary Key

The primary key is extracted from the entity and is used as a key in the Map. For example, if a Film entity has a primary key “film_id” that is of type int, then a corresponding Integer will be used as key. This is the recommended way for optimum performance.

Two or More Primary Keys

The primary keys are extracted from the entry and are put in a List that is used as a key in the Map. For example, if a FilmActor entity has a compound primary key consisting of the columns actor_id and film_id (both of type int), then a List containing two corresponding Integer objects will be used as key.

Portable Factories

The HazelcastToolBundle automatically generates the necessary PortableFactory objects. These are used to create Portable entities without resorting to reflection.
All PortableFactory classes are collected together in a single holder class and there is one PortableFactory for each schema in the database. For the Sakila sample database (that consist only of one schema also named “sakila”) the following classes are generated:

public final class SakilaPortableFactories {
    
    private SakilaPortableFactories() {}
    
    /**
     * A {@link PortableFactory } class for the schema sakila
     * <p>
     * This PortableFactory has an id of 1321754994
     */
    public final static class SakilaSakilaPortableFactory 
    extends GeneratedSakilaSakilaPortableFactory 
    implements PortableFactory {}
}

As can be seen, PortableFactory class just inherits all its methods from another generated class. This allows the possibility to override generated methods with custom code that is retained between re-generation of code.

@GeneratedCode("Speedment")
public final class GeneratedSakilaPortableFactories {
    
    private GeneratedSakilaPortableFactories() {}
    
    /**
     * A {@link PortableFactory } class for the schema sakila
     * <p>
     * This PortableFactory has an id of 1321754994
     */
    public abstract static class GeneratedSakilaSakilaPortableFactory implements PortableFactory {
        
        @Override
        public Portable create(int classId) {
            switch (classId) {
                case 92645877: return new ActorImpl();
                case 1147692044: return new AddressImpl();
                case 50511102: return new CategoryImpl();
                case 3053931: return new CityImpl();
                case 957831062: return new CountryImpl();
                case 606175198: return new CustomerImpl();
                case 3143044: return new FilmImpl();
                case 637325178: return new FilmActorImpl();
                case 205122905: return new FilmCategoryImpl();
                case 1087251704: return new FilmTextImpl();
                case 2020599460: return new InventoryImpl();
                case 1613589672: return new LanguageImpl();
                case 786681338: return new PaymentImpl();
                case 934576860: return new RentalImpl();
                case 109757152: return new StaffImpl();
                case 109770977: return new StoreImpl();
                case 1340382024: return new ActorInfoImpl();
                case 2143869857: return new CustomerListImpl();
                case 1087486343: return new FilmListImpl();
                case 970270416: return new NicerButSlowerFilmListImpl();
                case 47400380: return new SalesByFilmCategoryImpl();
                case 58908876: return new SalesByStoreImpl();
                case 260666013: return new StaffListImpl();
            }
            return null;
        }
    }
}

The PortableFactory objects are automatically added by the generated Configuration classes.

Class Definitions

The HazelcastToolBundle automatically generates ClassDefinition objects for each entity type. ClassDefinitiona are used by Hazelcast to internally define how the entity classes look like.
For the Sakila sample database table “film”, the following ClassDefinition objects are generated:

public class FilmClassDefinition extends GeneratedFilmClassDefinition {}

As can be seen, ClassDefinition class just inherits all its methods from another generated class. This allows the possibility to override generated methods with custom code that is retained between re-generation of code.

@GeneratedCode("Speedment")
public abstract class GeneratedFilmClassDefinition implements IntFunction<ClassDefinition> {
    
    protected GeneratedFilmClassDefinition() {}
    
    @Override
    public ClassDefinition apply(int version) {
        return new ClassDefinitionBuilder(1321754994, 3143044, version)
            .addIntField("film_id")
            .addUTFField("title")
            .addUTFField("description")
            .addLongField("release_year")
            .addBooleanField("__null__release_year")
            .addShortField("language_id")
            .addShortField("original_language_id")
            .addBooleanField("__null__original_language_id")
            .addShortField("rental_duration")
            .addUTFField("rental_rate")
            .addIntField("length")
            .addBooleanField("__null__length")
            .addUTFField("replacement_cost")
            .addUTFField("rating")
            .addUTFField("special_features")
            .addLongField("last_update")
            .build();
    }
}

The ClassDefinition objects are automatically added by the generated Configuration classes.

Supported Data Types

The following Java data types are supported:

  • byte, Byte
  • short, Short
  • int, Integer
  • long, Long
  • float, Float
  • double, Double
  • BigInteger
  • BigDecimal
  • Enum
  • boolean, Boolean
  • String
  • Timestamp
  • Time
  • Date
  • BLOB CLOB and Text columns are supported via String mapping.

For each column, there are a number of type mapping possibilities that can be applied using the UI Tool.

Null Handling

Via the UI Tool, nullable columns can be configured to use getters returning either null (i.e standard POJO) or Optional objects.

Configuration

The Hazelcast Auto DB Integration tool generates a complete class that provides a Hazelcast ClientConfiguration containing all serialization factories and class definitions already pre-configured. This class is named after the project name. For example, for a project named “Sakila”, then the configuration class will be named SakilaHazelcastConfigComponent. This is how an exemplary generated class looks like:

public class SakilaHazelcastConfigComponent extends GeneratedSakilaHazelcastConfigComponent {}

As can be seen, this class just inherits all its methods from another generated class. This allows the possibility to override generated methods with custom code that is retained between re-generation of code.

If you want to change the default configuration, there is a better way than overriding this class. Read about HazelcastConfigModifierComponent.

@GeneratedCode("Speedment")
public class GeneratedSakilaHazelcastConfigComponent implements HazelcastConfigComponent {
    
    protected GeneratedSakilaHazelcastConfigComponent() {}
    
    @Override
    public ClientConfig get() {
        final ClientConfig clientConfig = new ClientConfig();
        addPortableFactories(clientConfig);
        addClassDefinitions(clientConfig);
        return clientConfig;
    }
    
    protected void addPortableFactories(ClientConfig clientConfig) {
        clientConfig.getSerializationConfig()
            .addPortableFactory(1321754994, new SakilaSakilaPortableFactory())
        ;
    }
    
    protected void addClassDefinitions(ClientConfig clientConfig) {
        clientConfig.getSerializationConfig()
            .addClassDefinition(new ActorClassDefinition().apply(0))
            .addClassDefinition(new AddressClassDefinition().apply(0))
            .addClassDefinition(new CategoryClassDefinition().apply(0))
            .addClassDefinition(new CityClassDefinition().apply(0))
            .addClassDefinition(new CountryClassDefinition().apply(0))
            .addClassDefinition(new CustomerClassDefinition().apply(0))
            .addClassDefinition(new FilmClassDefinition().apply(0))
            .addClassDefinition(new FilmActorClassDefinition().apply(0))
            .addClassDefinition(new FilmCategoryClassDefinition().apply(0))
            .addClassDefinition(new FilmTextClassDefinition().apply(0))
            .addClassDefinition(new InventoryClassDefinition().apply(0))
            .addClassDefinition(new LanguageClassDefinition().apply(0))
            .addClassDefinition(new PaymentClassDefinition().apply(0))
            .addClassDefinition(new RentalClassDefinition().apply(0))
            .addClassDefinition(new StaffClassDefinition().apply(0))
            .addClassDefinition(new StoreClassDefinition().apply(0))
            .addClassDefinition(new ActorInfoClassDefinition().apply(0))
            .addClassDefinition(new CustomerListClassDefinition().apply(0))
            .addClassDefinition(new FilmListClassDefinition().apply(0))
            .addClassDefinition(new NicerButSlowerFilmListClassDefinition().apply(0))
            .addClassDefinition(new SalesByFilmCategoryClassDefinition().apply(0))
            .addClassDefinition(new SalesByStoreClassDefinition().apply(0))
            .addClassDefinition(new StaffListClassDefinition().apply(0))
        ;
    }
}

Thus, the generated configuration class adds all the portable serialization factories and all class definitions that has been automatically generated. This class is automatically added as a component by the application builder.

Custom Configuration

Custom configuration can be injected using any class(es) that implements HazelcastConfigModifierComponent as examplified hereunder:

    public class MyHazelcastConfigModifierComponent implements HazelcastConfigModifierComponent {

        @Override
        public ClientConfig apply(ClientConfig clientConfig) {
            System.out.println("My custom changes were applied");
            clientConfig.getNetworkConfig().addAddress("192.168.0.234:8234");
            return clientConfig;
        }
    }

This class is then injected into the application builder as shown here:

   final Speedment hazelcastApp = new SakilaApplicationBuilder()
            .withPassword("sakila-password")
            .withBundle(HazelcastBundle.class)
            .withComponent(MyHazelcastConfigModifierComponent.class)
            .build();

This will add the address 192.168.0.234:8234 to the Hazelcast client’s network address and will print:

My custom changes were applied

Any number of HazelcastConfigModifierComponent classes may be added to the builder and they will be applied in order of injection.

Ingesting Data via a Client

Ingest of data from the database into the Hazelcast cluster can be made in many ways.

Generated Ingest Main Method

The ‘HazelcastToolBundle’ generates a default ingest main method that can be used to ingest data as depicted below:

public final class SakilaIngest {
    
    public static void main(final String... argv) {
        if (argv.length == 0) { 
            System.out.println("Usage: " + SakilaIngest.class.getSimpleName() + " database_password");
         } else {
            try (Speedment app = new SakilaApplicationBuilder()
                .withPassword(argv[0]) // Get the password from the first command line parameter
                .withBundle(HazelcastBundle.class)
                .build()) {
            
                IngestUtil.ingest(app).join();
            }
        }
    }
}

Just calling this main method with a single command line argument with the password of the database will take care of the entire ingest procedure.

Custom Ingest Methods

Ingesting data from a database into the Hazelcast server nodes is greatly simplified with a provided utility class named IngestUtil. The following example shows a method that will invoke a method IngestUtil::ingest to ingest data from all tables in the the database into the Hazelcast server grid:

    public void ingestAll() {

        // Create a Speedment application connected to a SQL database
        // and that also contains a Hazelcast client
        final Speedment app = new SakilaApplicationBuilder()
            .withPassword("sakila-password")
            .withBundle(HazelcastBundle.class)
            .build();

        // Ingest all tables from the database into the Hazelcast
        // server grid using the default IngestConfiguration:
        // - Load all data (i.e. all rows) from the tables
        // - Use the default ForkJoin pool for parallel loading
        // - Perform loading outside any database transaction
        // - Do not clear the maps before loading
        CompletableFuture<Void> job = IngestUtil.ingest(app);

        // Wait for the ingest job to complete
        job.join();

        // Print out att the distributed maps that now has been
        // created and populated with data
        app.getOrThrow(HazelcastInstanceComponent.class).get()
            .getDistributedObjects()
            .stream()
            .sorted(comparing(Object::toString))
            .forEach(System.out::println);

        // Close the app and thereby also the hazelcastInstance
        app.close();

    }

This produces the following output showing all the IMap objects in which data was ingested:

FlakeIdGenerator{name='sakila.sakila.actor_info'}
FlakeIdGenerator{name='sakila.sakila.customer_list'}
FlakeIdGenerator{name='sakila.sakila.film_list'}
FlakeIdGenerator{name='sakila.sakila.nicer_but_slower_film_list'}
FlakeIdGenerator{name='sakila.sakila.sales_by_film_category'}
FlakeIdGenerator{name='sakila.sakila.sales_by_store'}
FlakeIdGenerator{name='sakila.sakila.staff_list'}
IMap{name='sakila.sakila.actor'}
IMap{name='sakila.sakila.actor_info'}
IMap{name='sakila.sakila.address'}
IMap{name='sakila.sakila.category'}
IMap{name='sakila.sakila.city'}
IMap{name='sakila.sakila.country'}
IMap{name='sakila.sakila.customer'}
IMap{name='sakila.sakila.customer_list'}
IMap{name='sakila.sakila.film'}
IMap{name='sakila.sakila.film_actor'}
IMap{name='sakila.sakila.film_category'}
IMap{name='sakila.sakila.film_list'}
IMap{name='sakila.sakila.film_text'}
IMap{name='sakila.sakila.inventory'}
IMap{name='sakila.sakila.language'}
IMap{name='sakila.sakila.nicer_but_slower_film_list'}
IMap{name='sakila.sakila.payment'}
IMap{name='sakila.sakila.rental'}
IMap{name='sakila.sakila.sales_by_film_category'}
IMap{name='sakila.sakila.sales_by_store'}
IMap{name='sakila.sakila.staff'}
IMap{name='sakila.sakila.staff_list'}
IMap{name='sakila.sakila.store'}

Note: The FlakeIdGenerator objects are used for tables/views that have no primary key.

The utility class IngestUtil contains a number of related methods that can be used to control the ingest process in more detail, including:

  • Selecting a custom ExecutorService used to ingest data
  • Selecting a database transaction to use during data ingest
  • Applying arbitrary Stream operators on the database source Stream (e.g. limiting or filtering the database content)
  • Clearing all data before start of data ingest
  • Selecting a subset of database tables to use during ingest

See the JavaDoc for the classes IngestUtil and IngestConfiguration for a detailed description on these features.

The code above can be shortened like this (provided that the print out section is skipped):

    public void ingestAllShort() {

        try (Speedment app = new SakilaApplicationBuilder()
            .withPassword("sakila-password")
            .withBundle(HazelcastBundle.class)
            .build()) {

            IngestUtil.ingest(app).join();
        };

    }

Query Data

Data can be queried using at least three different methods:

  • Hazelcast IMap API
  • Hazelcast Jet (distributed streams)
  • Standard Java Streams

Hazelcast Map

Applications can use the Hazelcast Map and IMap interfaces directly and work with data this way. The name of a distributed map can be obtained using the HazelcastMapUtil::mapName method as shown hereunder:

String filmMapName = HazelcastMapUtil.mapName(FilmManager.IDENTIFIER); 

Read more on the Hazelcast IMap API here

Hazelcast Jet

Read more on connecting Hazelcast Jet to Hazelcast IMap objects here

Streams

As for all Speedment applications, data in the Hazelcast grid can be queried using standard java.util.stream.Stream objects. Here is an example where we are collecting a list of the film titles (in alphabetical order) of the films with a rating of PG-13 that has a length greater or equal to 180 minutes:

FilmManager films = hazelcastApp.getOrThrow(FilmManager.class);

List<String> list = films.stream()
    .filter(Film.RATING.equal("PG-13"))
    .filter(Film.LENGTH.greaterOrEqua(180))
    .map(Film.TITLE)
    .sorted()
    .collect(Collectors.toList());

Read more about Speedment streams here (examples) and here (stream fundamentals)

Other Languages

Because data is stored using Portable entity classes, data in the Hazelcast server nodes can also be queried using other languages. Read more here.

Persistence

The standard Speedment CRUD operations API apply to Hazelcast in the same way as for other data sources. Please refer to that section of the user guide for examples of Speedment powered CRUD operations.

A Speedment application configured with a Hazelcast bundle will provide managers that handle persistence to both the Hazelcast grid and the underlying database making sure the underlying database serves as source of truth and that the Hazelcast data grid is eventually consistent with that source of truth.

By persisting to both, the Speedment runtime maintains the consistency between the relational database and the data grid. This of course assumes that only Speedment applications perform data altering operations on the data grid.

Persisting to Hazelcast Grid only

The Speedment application can be configured to operate on the Hazelcast grid only and not persist any data to the relational database by setting the ‘hazelcast.writethrough’ parameter to “false” as follows.

        Speedment hazelcastApp = new SakilaApplicationBuilder()
            .withPassword("sakila-password")
            .withBundle(HazelcastBundle.class)
            .withParam("hazelcast.writethrough", "false")
            .build();

Persisting Server-Side via MapStore/MapLoader

Instead of using persistence via a client, it is also possible to use server-side persistence via the MapStore/MapLoad interfaces. There are different pros and cons of this compared to Client based persistence.

Pros

  • Server nodes can load entities that belongs to its partitions directly
  • Server nodes can load entities in parallel, independent of other server nodes
  • Hazelcast maps can be loaded lazilly.

Cons

  • The Java Domain Model needs to be present on the server classpath
  • The server nodes need to know the database password
  • The server nodes need to be able to access the database

The HazelcastToolBundle automatically generates MapStore objects for each entity type:

public class FilmMapStore extends GeneratedFilmMapStore {
    
    public FilmMapStore(Manager<Film> manager) {
        super(manager);
    }
}

As can be seen, an entity implementation class just inherits all its methods from another generated class. This allows the possibility to override generated methods with custom code that is retained between re-generation of code.

@GeneratedCode("Speedment")
public abstract class GeneratedFilmMapStore extends AbstractMapStore<Integer, Film> {
    
    protected GeneratedFilmMapStore(Manager<Film> manager) {
        super(manager, Film.FILM_ID, new FilmImpl()::setFilmId);
    }
}

In order to initialize a FilmMapStore we need a Manager<Film> that can be retrieved directly from a Speedment instance as shown in the following example:

    public static final String MAP_NAME = HazelcastMapUtil.mapName(FilmManager.IDENTIFIER);

    public static void main(String... args) {

        final Speedment speedment = new SakilaApplicationBuilder()
            .withPassword("sakila-password")
            .withBundle(HazelcastServerBundle.class) // Use this Bundle server-side
            .build();

        // Creates a config with pre-configured serialization factories an more
        final Config config = new SakilaHazelcastServerConfigComponent().get();

        MapStoreConfig mapStoreConfig = new MapStoreConfig();
        mapStoreConfig.setImplementation(new FilmMapStore(speedment.getOrThrow(FilmManager.class)));
        mapStoreConfig.setWriteDelaySeconds(0);
        mapStoreConfig.setInitialLoadMode(MapStoreConfig.InitialLoadMode.EAGER);

        MapConfig mapConfig = config.getMapConfig(MAP_NAME);
        mapConfig.setMapStoreConfig(mapStoreConfig);

        final HazelcastInstance instance = Hazelcast.newHazelcastInstance(config);

        IMap<Integer, Film> map = instance.getMap(MAP_NAME);

        System.out.println("map.size() = " + map.size());
        
        instance.shutdown();
        speedment.close();
        
    }

This will produce the following output:

map.size() = 1000

As can be seen, the film map was pre-loaded by the server using the generated FilmMapLoader.

Both write-through and write-back modes are supported.

Transactions

Speedment transaction handling is further described here and applies to the relational database in the same way when using the Hazelcast bundle. In the current version of the Hazelcast bundle, the operations on the Hazelcast data grid are not covered by transactional locks. This is likely to change in some future release of the Hazelcast bundle where also operations on the data grid may support transactions.

Indexing

Upon generation, the Hazelcast Auto DB Integration tool examines the database metadata and suggests indexing based on how the database is indexed. This provides a solid baseline for grid indexing. In the following example, an index utility method was automatically generated when working with the Sakila database (the class has been shortened for brevity):

@GeneratedCode("Speedment")
public final class GeneratedSakilaIndexUtil {
    
    private GeneratedSakilaIndexUtil() {}
    
    public static void setupIndex(final HazelcastInstance h) {
        
        // Indexes for table actor
        // Index PRIMARY (unique) using column actor_id
        h.getMap("sakila.sakila.actor").addIndex("actor_id", true);
        // Index idx_actor_last_name  using column last_name
        h.getMap("sakila.sakila.actor").addIndex("last_name", true);
        
        // ... Rows hidden for brevity
        
        // Indexes for table film
        // Index PRIMARY (unique) using column film_id
        h.getMap("sakila.sakila.film").addIndex("film_id", true);
        // Index idx_film_rating  using column rating
        h.getMap("sakila.sakila.film").addIndex("rating", true);
        // Index idx_fk_language_id  using column language_id
        h.getMap("sakila.sakila.film").addIndex("language_id", true);
        // Index idx_fk_original_language_id  using column original_language_id
        h.getMap("sakila.sakila.film").addIndex("original_language_id", true);
        // Index idx_title  using column title
        h.getMap("sakila.sakila.film").addIndex("title", true);
        
        // ... Rows hidden for brevity
    }
}

As can be seen, creating a HazelcastInstance and then just invoking the method GeneratedSakilaIndexUtil::setupIndex will create the same indexes in the Hazelcast grid that were present in the database.

Joins

Hazelcast tables can be joined using the Stream Join functionality. The current Hazelcast version does not support joining of Hazelcast Maps using the JoinComponent. Read more about joins with Speedment here.

Aggregations

Aggregations using the Speedment Aggregator are supported with Hazelcast maps but are not fully optimized in the current version.

Aggregations using the Hazelcast IMap::aggregate method is fully supported.

Performance

Thanks to the Portable entity classes, Hazelcast server nodes can benefit from indexing and partial deserialization when applying predicates on large data sets. This greatly speeds up querying in many cases.

Example Clients

The following examples show different variants of Hazelcast clients that can connect to a Hazelcast server grid that already contains data (e.g. by means of the ingest feature):

Using the Hazelcast IMap API

This example is using the native Hazelcast IMap interface to select data from a Hazelcast server grid:

public class IMapClientExample {

    public static void main(String... args) {

        // Create the Speedment instance with the HazelcastBundle
        Speedment hazelcastApp = new SakilaApplicationBuilder()
            .withPassword("sakila-password")
            .withBundle(HazelcastBundle.class)
            .build();

        // Retrieve the HazecastInstance from the app
        HazelcastInstance hazelcastClient = hazelcastApp.getOrThrow(HazelcastInstanceComponent.class).get();

        // Calculate the IMap name given a TableIdentifier
        String mapName = mapName(FilmManager.IDENTIFIER);
        IMap<Integer, Film> filmMap = hazelcastClient.getMap(mapName);

        Predicate predicate = new SqlPredicate("rating = 'PG-13' and length >= 180");
        Collection<Film> collection = filmMap.values(predicate);

        // print out all Film entities that matches the predicates
        collection
            .forEach(System.out::println);

        // Close the hazelcastApp which, in turn, closes the hazelcastClient
        hazelcastApp.close();

    }

}

The code above can be shortened like this:

        try(Speedment app = new SakilaApplicationBuilder()
            .withPassword("sakila-password")
            .withBundle(HazelcastBundle.class)
            .build()) {
            
            app.getOrThrow(HazelcastInstanceComponent.class)
                .get()
                .getMap(mapName(FilmManager.IDENTIFIER))
                .values(new SqlPredicate("rating = 'PG-13' and length >= 180"))
                .forEach(System.out::println);
        }

Using the Speedment Stream API

This example is using the Speedment Stream API to select data from a Hazelcast server grid:

public class StreamClientExample {

    public static void main(String... args) {

        // Create the Speedment instance with the HazelcastBundle
        Speedment hazelcastApp = new SakilaApplicationBuilder()
            .withPassword("sakila-password")
            .withBundle(HazelcastBundle.class)
            .build();

        // Retrieve the FilmManager from the app
        FilmManager films = hazelcastApp.getOrThrow(FilmManager.class);

        // Print out all Film entities that matches the predicates
        films.stream()
            .filter(Film.RATING.equal("PG-13"))
            .filter(Film.LENGTH.greaterOrEqual(180))
            .forEach(System.out::println);

        // Close the hazelcastApp which, in turn, closes the hazelcastClient
        hazelcastApp.close();

    }

}

The code above can be shortened like this:

        try (Speedment app = new SakilaApplicationBuilder()
            .withPassword("sakila-password")
            .withBundle(HazelcastBundle.class)
            .build()) {

            app.getOrThrow(FilmManager.class).stream()
                .filter(Film.RATING.equal("PG-13"))
                .filter(Film.LENGTH.greaterOrEqual(180))
                .forEach(System.out::println);

        }

Hazelcast cloud

Hazelcast Auto DB Integration verison 3.1.14 and later are ready for usage in Hazelcast cloud. A cloud instance is defined by three parameters - the name, the group password and the discovery token. These three are supplied to Hazelcast Auto DB Integration via a HazelcastCloudConfig as follows.

        Speedment hazelcastApp = new SakilaApplicationBuilder()
            .withPassword("sakila-password")
            .withBundle(HazelcastBundle.class)
            .withComponent(HazelcastCloudConfig.class, () -> HazelcastCloudConfig.create(
                            "<name of cluster>",
                            "<cluster password>",
                            "<discovery token>")
            )
            .build();

Discussion

Join the discussion in the comment field below or on Gitter