* [Kotlin] Small optimizations and benchmark on deserialization
* [Kotlin] Remove redudant assign() method (use init() instead)
* [Kotlin] Fix benchmark run after change in flatbuffers-java deps
Commit 6e214c3a49 fixes Kotlin build,
but makes the kotlin-benchmark plugin misses the java classes at
runtime, causing NotClassFoundError. The alternative to solve the issue
is to read java's pom.xml to get the latest java version and use it
as dependency. With that we avoid compilation errors on a new version and
keep benchmark plugin happy.
By using specific jar version to use java's runtime on the benchmark
module we let CI break when new versions are released. So we are using
source directly instead
* [Kotlin] Introduction to Kotlin Multiplaform
The first implementation of the Kotlin code generation was made years
ago at the time Kotlin Multiplaform was not stable and Kotlin is mostly
used on JVM-based targets. For this reason the generated code uses java
based runtime.
That design decision comes with many drawbacks, leaving the code
generated more java-like and making it impossible to use more advanced
features of the Kotlin language.
In this change we are adding two parts: A pure, multi-plaform, Kotlin
runtime and a new code generator to accompany it.
* [Kotlin] Remove scalar sign cast from code generation
Now that we have a new runtime the accepts unsigned types, we don't
need to code generate casting back and from signed scalars. This
MR removes this from both code generations and adds the necessary
API to the runtime.
* [Kotlin] Use offset on public API to represent buffer position
Currently, kotlin was following Java's approach of representing objects,
vectors, tables as "Int" (the position of it in the buffer). This change
replaces naked Int with Offset<T>, offering a type-safe API. So,
instead of
fun Table.createTable(b: FlatBufferBuilder, subTable: Int)
We will have
fun Table.createTable(b: FlatBufferBuilder, subTable: Offset<SubTable>)
Making impossible to accidentally switch parameters.
The performance should be similar to use Int as we are using value
class for Offset and ArrayOffset, which most of the time translate to
Int in the bytecode.
* [Kotlin] Add builder for tables
Add builder constructor to make create of table more ergonomic.
For example the movie sample for the test set could be written as:
Movie.createMovie(fbb,
mainCharacterType = Character_.MuLan,
mainCharacter = att) {
charactersType = charsType
this.characters = characters
}
instead of:
Movie.startMovie(fbb)
Movie.addMainCharacterType(fbb, Character_.MuLan)
Movie.addMainCharacter(fbb, att as Offset<Any>)
Movie.addCharactersType(fbb, charsType)
Movie.addCharacters(fbb, charsVec)
Movie.endMovie(fbb)
* [Kotlin] Move enum types to value class
Moving to flatbuffer enums to value class adds type safety for parameters
with minimum to no performance impact.
* [Kotlin] Simplify Union parameters to avoid naked casting
Just a small change on the APIs that receive union as parameters,
creating a typealias UnionOffset to avoid using Offset<Any>. To "convert"
an table offset to an union, one just call Offset.toUnion().
* [Kotlin] Apply clang-format on kotlin code generators
* [Kotlin] Update kotlin generator to follow official naming conventions
Updating directory, package and enum naming to follow Kotlin official
convention.
https://kotlinlang.org/docs/coding-conventions.html#naming-rules
* [Kotlin] Add fixes to improve performance
1 - Add benchmark comparing serialization between Java & Kotlin
2 - ReadWriteBuffer does not auto-grow (thus avoid check size in every op)
3 - Add specialized add functions on FlatBufferBuilder to avoid boxing
offsets.
4 - Remove a few Kotlin syntax sugar that generated performance penalties.
* [Kotlin] Remove builder from Kotlin KMP and add some optimizations
to avoid boxing of Offset classes
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
ToCamelCase(input, true) converts first char to upper case, but
ToCamelCase(input, false) keeps the case of the first char. We are
changing its behavior to force a lower case.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* [Android] fixed build after decomission of jcenter
JCenter[1] has been removed and now is failing android build. This
change updates the configuration to remove this and few other warnings.
1 - https://developer.android.com/studio/build/jcenter-migration
* [Kotlin] fix build for latest gradle version 8.0.1
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
To make it simple to map between a union field and its union type
field we are adding a pointer to FieldDef to point to each other. For
all other types the pointer will be nullptr.
Co-authored-by: Derek Bailey <derekbailey@google.com>
Untyped fixed vectors are not supported in FlexBuffers. There
was an assert to check for it, but on java, asserts are optional. This
change converts the assertion into a runtime exception.
Fixes#7358
Co-authored-by: Derek Bailey <derekbailey@google.com>
With a change introduce in 385dda5c3785ed8d6a35868bc169f07e40e889087fd2edc66,
flatc was not able to emit code for Kotlin if a namespace is specified
and the folders do not exist. The fix create folders if neded.
Additional changes are introduced in gradle files to bring more visibility
to the error messages.
Co-authored-by: Derek Bailey <derekbailey@google.com>
There was a step in the compilation process where benchmark data is
downloaded before starting the kotlin compilation process.
Since we are not running benchmark on CI anymore, we remove the
dependency. To run benchmarks the download task needs to be executed
manually.
* [Java] Fix key lookup returning null clashing with default value
A field with key attribute must always be written on the message so it
can be looked up by key. There is a edge case where inserting a key
field with same value as default would prevent it to be written on
the message and later cannot be found when searched by key.
* [Kotlin] Fix key lookup returning null clashing with default value
A field with key attribute must always be written on the message so it
can be looked up by key. There is a edge case where inserting a key
field with same value as default would prevent it to be written on
the message and later cannot be found when searched by key.
Co-authored-by: Derek Bailey <derekbailey@google.com>
A field with key attribute must always be written on the message so it
can be looked up by key. There is a edge case where inserting a key
field with same value as default would prevent it to be written on
the message and later cannot be found when searched by key.
* [Kotlin] Update gradle to 7.4.1 and simplify config files.
* [Kotlin] Add wrapper-validation-action to Kotlin Linux
* [Kotlin] Remove benchmark actions to reduce CI time
* [Kotlin] Move CI js test to Linux action, to increase Mac action speed
* [Kotlin] Generate gradle wrapper in order to be validate
Gradle wrapper from 3.3 to 4.0 are not verifiable because those files
were dynamically generated by Gradle in a non-reproducible way.
So they are now regenerated and will be checked using gitlab action:
gradle/wrapper-validation-action@v1
With the changes introduced on #6729, #6671, #6658 it seems that using java
compiler version 8 is no longer possible. We are adjusting our CI build for
Kotlin to use Java 11 as compiler, while still emiting 1.8 bytecode.
The Kotlin CI action is also being breakdown into two: Mac & Linux.
Kotlin mac will test ios & mac builds while Linux will test js and JVM.
This change will improve build times for Kotlin on CI, which is currently
taking long times.
on the current Java Implementation.
The code dependencies related to JVM were removed and the project is able to target all available platforms.
The only requirement to implement to fully support a target is to implement functions described in `ByteArray.kt`.
Right now the code support JVM and native targets. JS port still missing, but just be trivial to introduce.
Currently, only the `jvm` and `macosX64` targets are enabled until we figure out how to enable tests on all
platforms on CI.
A submodule called "benchmark" is also introduced. It contains a series
of benchmarks comparing Java and Kotlin implementations of FlexBuffers and the UTF8 API.
Finally, this commit does not contain the scripts necessary to publish the artifacts. This will
be introduced at a later stage once the team has an agreement on how to rollout Kotlin releases.
Android build was dated, using the Android.mk approach. Current
project configuration on Android encourages the usage of CMake, so we
are updating the android project as an example on how to use either the
Java/Kotlin generate code or the native C++ one.
* Add test for optional enums in Kotlin
* Rename optional_scalars2.fbs into optional_scalars.fbs
Also updated all references in the project to point to
"optional_scalars.fbs" instead of "optional_scalars2.fbs".
Added a method FlatBufferBuilder::createSharedString that
enable string sharing for building messages on java.
The shared pool will only contains strings inserted by
this methods.
Kotlin code generation was producing wrong logic for accessors
of vector of union elements. This was shadowed by the fact[1] that asserts
in Kotlin are silently ignored unless the flag "-ea" is passed to the JVM.
The tests are also updated to enable asserts.
1 - https://kotlinlang.org/api/latest/jvm/stdlib/kotlin/assert.html
The original implementation of map access is very naive:
- Encode String to UTF8 byte[]
- Creates a new KeyVector
- Performs a binary search to find the key
- return value
So every access to the Map there was useless allocations of Keys and KeyVector
and complete encoding of the search key, which for most comparisons would be wasteful.
This changes completely removes the use of KeyVector and compute the key
positions on the spot. Besides that, it compares keys codepoint-by-codepoint,
avoiding unnecessary allocations and reducing encoding for most cases.
Some benchmarks result in a 2.75x speedup.
To read and build flexbuffers on Java, one needs to wrap the data
using ByteBuffer. But for the common case of having ByteBuffers
backed by arrays, accessing from a ByteBuffer might be inefficient.
So this change introduces two interfaces: ReadBuf and ReadWriteBuf.
It allows one to read and writes data directly on an array. It also allow
other buffer implementations to be used with flexbuffers.
Another change is that FlexBuffersBuilder backed by array allows
the buffer to grow with the increase of the message size. Something
that could not be done with ByteBuffer.
* [Java][FlexBuffers] Deprecate typed vector strings due to design flaw
It will still be possible to read buffers with this type, but the
elements will be treated as FBT_KEY and will be read as null-terminated
string.
Trying to build a vector of strings as typed will throw an exception.
More information on https://github.com/google/flatbuffers/issues/5627
Also, fix another bug on strings, where long strings were not properly
aligned.
* [Java][FlexBuffers] Make FBT_VECTOR_STRING_DEPRECATED considered typed.
The logic for FlexBuffers.isVectorType() was changed
to not consider FBT_VECTOR_STRING_DEPRECATED a typed
vector, but that can lead to missinterpretation for
existing serialized data. So we are reverting.
Kolin uses java library as dependency, which changed the way it access union vector recently
(e365c502ff).
This changes updates kotlin code generation to match Java's changes.
The condition was unnecessary and Detected by
PVS-Studio
V560 [CWE-571] A part of conditional expression is always true: !opts.use_flexbuffers. flatc.cpp 438
Empty objects that inherit from Sized would try to access internal
ByteBuffer when Sized::size was called. So we add a single byte in
the empty buffer, so when size() is called it would return 0
The real position of a string is calculated by using the indirect() method,
which should be based on parentWidth and not byteWidth, as it was implemented.
We are also fixing the flag BUILDER_FLAG_SHARE_STRINGS on FlexBuffersBuilder
that was set as '1', same value as BUILDER_FLAG_SHARE_KEYS.
* [FlexBuffers][Java] Implementation of FlexBuffers API
This is the initial attemp to implement FlexBuffer on Java.
There is some limitations as compared to the C++ implementation:
1 - No mutations implemented yet
2 - Does not parse from json
Also, this initial implementation is not focused and performance, but
get the basics write. So there is many opportunities for optimization, for instance,
remove all enums, return CharSequence instead of Strings and object pooling.
* [FlexBuffers][Java] Optimizations and simplification of the Builder API.
This change removes BitWidth enum in favor of static ints. Also
make all "reads" APIs closer to C++ implementation (try to cast or convert
as much as possible, assuming user knows what he is doing). Finally,
we remove the helper classes for building vectors and maps.
There is no official benchmarks, but the unit tests are running in less
than 50% for previous runs, which mean those optimizations are worth it.
* [FlexBuffers][Java] Fix Reference::asString behavior
There was a incorrect assumption that strings would be null-terminated, which
could lead to truncated strings. S now it relies size instead of null-termination.
Other minor improvements
* [Kotlin] Add kotlin generate code for tests and add
kotlin test to TestAll.sh
* [Kotlin] Add Kotlin generator
This change adds support for generating Kotlin classes.
The approach of this generator is to keep it as close
as possible to the java generator for now, in order
to keep the change simple.
It uses the already implemented java runtime,
so we don't support cross-platform nor js Kotlin yet.
Kotlin tests are just a copy of the java tests.
* Add optional ident support for CodeWriter
Identation is important for some languages and
different projects have different ways of ident
code, e.g. tabs vs spaces, so we are adding optional
support on CodeWriter for identation.
* [Kotlin] Add Documentation for Kotlin
* [Kotlin] Modify generated code to use experimental Unsigned types.
