* [Kotlin] Small optimizations and benchmark on deserialization
* [Kotlin] Remove redudant assign() method (use init() instead)
* [Kotlin] Fix benchmark run after change in flatbuffers-java deps
Commit 6e214c3a49 fixes Kotlin build,
but makes the kotlin-benchmark plugin misses the java classes at
runtime, causing NotClassFoundError. The alternative to solve the issue
is to read java's pom.xml to get the latest java version and use it
as dependency. With that we avoid compilation errors on a new version and
keep benchmark plugin happy.
* [Kotlin] Introduction to Kotlin Multiplaform
The first implementation of the Kotlin code generation was made years
ago at the time Kotlin Multiplaform was not stable and Kotlin is mostly
used on JVM-based targets. For this reason the generated code uses java
based runtime.
That design decision comes with many drawbacks, leaving the code
generated more java-like and making it impossible to use more advanced
features of the Kotlin language.
In this change we are adding two parts: A pure, multi-plaform, Kotlin
runtime and a new code generator to accompany it.
* [Kotlin] Remove scalar sign cast from code generation
Now that we have a new runtime the accepts unsigned types, we don't
need to code generate casting back and from signed scalars. This
MR removes this from both code generations and adds the necessary
API to the runtime.
* [Kotlin] Use offset on public API to represent buffer position
Currently, kotlin was following Java's approach of representing objects,
vectors, tables as "Int" (the position of it in the buffer). This change
replaces naked Int with Offset<T>, offering a type-safe API. So,
instead of
fun Table.createTable(b: FlatBufferBuilder, subTable: Int)
We will have
fun Table.createTable(b: FlatBufferBuilder, subTable: Offset<SubTable>)
Making impossible to accidentally switch parameters.
The performance should be similar to use Int as we are using value
class for Offset and ArrayOffset, which most of the time translate to
Int in the bytecode.
* [Kotlin] Add builder for tables
Add builder constructor to make create of table more ergonomic.
For example the movie sample for the test set could be written as:
Movie.createMovie(fbb,
mainCharacterType = Character_.MuLan,
mainCharacter = att) {
charactersType = charsType
this.characters = characters
}
instead of:
Movie.startMovie(fbb)
Movie.addMainCharacterType(fbb, Character_.MuLan)
Movie.addMainCharacter(fbb, att as Offset<Any>)
Movie.addCharactersType(fbb, charsType)
Movie.addCharacters(fbb, charsVec)
Movie.endMovie(fbb)
* [Kotlin] Move enum types to value class
Moving to flatbuffer enums to value class adds type safety for parameters
with minimum to no performance impact.
* [Kotlin] Simplify Union parameters to avoid naked casting
Just a small change on the APIs that receive union as parameters,
creating a typealias UnionOffset to avoid using Offset<Any>. To "convert"
an table offset to an union, one just call Offset.toUnion().
* [Kotlin] Apply clang-format on kotlin code generators
* [Kotlin] Update kotlin generator to follow official naming conventions
Updating directory, package and enum naming to follow Kotlin official
convention.
https://kotlinlang.org/docs/coding-conventions.html#naming-rules
* [Kotlin] Add fixes to improve performance
1 - Add benchmark comparing serialization between Java & Kotlin
2 - ReadWriteBuffer does not auto-grow (thus avoid check size in every op)
3 - Add specialized add functions on FlatBufferBuilder to avoid boxing
offsets.
4 - Remove a few Kotlin syntax sugar that generated performance penalties.
* [Kotlin] Remove builder from Kotlin KMP and add some optimizations
to avoid boxing of Offset classes
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
* First working hack of adding 64-bit. Don't judge :)
* Made vector_downward work on 64 bit types
* vector_downward uses size_t, added offset64 to reflection
* cleaned up adding offset64 in parser
* Add C++ testing skeleton for 64-bit
* working test for CreateVector64
* working >2 GiB buffers
* support for large strings
* simplified CreateString<> to just provide the offset type
* generalize CreateVector template
* update test_64.afb due to upstream format change
* Added Vector64 type, which is just an alias for vector ATM
* Switch to Offset64 for Vector64
* Update for reflection bfbs output change
* Starting to add support for vector64 type in C++
* made a generic CreateVector that can handle different offsets and vector types
* Support for 32-vector with 64-addressing
* Vector64 basic builder + tests working
* basic support for json vector64 support
* renamed fields in test_64bit.fbs to better reflect their use
* working C++ vector64 builder
* Apply --annotate-sparse-vector to 64-bit tests
* Enable Vector64 for --annotate-sparse-vectors
* Merged from upstream
* Add `near_string` field for testing 32-bit offsets alongside
* keep track of where the 32-bit and 64-bit regions are for flatbufferbuilder
* move template<> outside class body for GCC
* update run.sh to build and run tests
* basic assertion for adding 64-bit offset at the wrong time
* started to separate `FlatBufferBuilder` into two classes, 1 64-bit aware, the other not
* add test for nested flatbuffer vector64, fix bug in alignment of big vectors
* fixed CreateDirect method by iterating by Offset64 first
* internal refactoring of flatbufferbuilder
* block not supported languages in the parser from using 64-bit
* evolution tests for adding a vector64 field
* conformity tests for adding/removing offset64 attributes
* ensure test is for a big buffer
* add parser error tests for `offset64` and `vector64` attributes
* add missing static that GCC only complains about
* remove stdint-uintn.h header that gets automatically added
* move 64-bit CalculateOffset internal
* fixed return size of EndVector
* various fixes on windows
* add SizeT to vector_downward
* minimze range of size changes in vector and builder
* reworked how tracking if 64-offsets are added
* Add ReturnT to EndVector
* small cleanups
* remove need for second Array definition
* combine IndirectHelpers into one definition
* started support for vector of struct
* Support for 32/64-vectors of structs + Offset64
* small cleanups
* add verification for vector64
* add sized prefix for 64-bit buffers
* add fuzzer for 64-bit
* add example of adding many vectors using a wrapper table
* run the new -bfbs-gen-embed logic on the 64-bit tests
* remove run.sh and fix cmakelist issue
* fixed bazel rules
* fixed some PR comments
* add 64-bit tests to cmakelist
* Add binary schema reflection
* remove not-used parameter
* move logic from object API to base API
* forward declare
* remove duplicate code gen that was stompping on the edits
* reduce to just typedef generation
* fixed bazel rules to not stomp
* more bazel fixes to support additional generated files
* Migrate from rules_nodejs to rules_js/rules_ts (take 2)
This is the second version of patch #7923. The first version got
reverted because bazel query was failing:
$ bazel --nosystem_rc --nohome_rc query tests(set('//...')) except tests(attr("tags", "manual", set('//...')))
ERROR: Traceback (most recent call last):
File "/workdir/tests/ts/bazel_repository_test_dir/BUILD", line 6, column 22, in <toplevel>
npm_link_all_packages(name = "node_modules")
File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/external/npm/defs.bzl", line 188, column 13, in npm_link_all_packages
fail(msg)
Error in fail: The npm_link_all_packages() macro loaded from @npm//:defs.bzl and called in bazel package 'tests/ts/bazel_repository_test_dir' may only be called in bazel packages that correspond to the pnpm root package '' and pnpm workspace projects ''
This was happening because the `.bazelrc` file only added
`--deleted_packages` to the `build` command. We also need it for the
`query` command. This second version of the patch fixes that.
Original commit message:
This patch migrates the current use of rules_nodejs to the new rules_js.
rules_js is the intended replacement of rules_nodejs as per this note:
https://github.com/aspect-build/rules_js#relationship-to-rules_nodejs
> rules_js is an alternative to the build_bazel_rules_nodejs Bazel module
> and accompanying npm packages hosted in
> https://github.com/bazelbuild/rules_nodejs, which is now
> unmaintained. All users are recommended to use rules_js instead.
There are a few notable changes in this patch:
1. The `flatbuffer_ts_library` macro no longer accepts a `package_name`
attribute. This is because rules_js appears to manage the import
naming of dependencies via top-level `npm_link_package` targets.
Users will have to migrate.
2. I added a few more arguments to `flatbuffer_library_public()`. These
helped with exposing esbuild to `ts/compile_flat_file.sh`.
3. I pinned the version of `typescript` in `package.json` so that
rules_ts can download the exact same version. rules_ts doesn't know
what to do if the version isn't exact.
4. Since rules_js uses the pnpm locking mechanism, we now have a
`pnpm-lock.yaml` file instead of a yarn lock file.
4. I added bazel targets for a few of the existing tests in `tests/ts`.
They can be run with `bazel test //test/ts:all`. Since there is no
flexbuffers bazel target, I did not add a bazel target for the
corresponding test.
5. I added a separate workspace in `tests/ts/bazel_repository_test_dir/`
to validate that the flatbuffers code can be imported as an external
repository. You can run the test with
`bazel test //test/ts:bazel_repository_test`. For this to work, I
needed to expose a non-trivial chunk of the flatbuffers code to the
test. I achieved this through some recursive `distribution`
filegroups. This is inspired by rules_python's workspace tests.
I did not do anything special to validate that the `gen_reflections`
parameter works the same. This patch doesn't change anything about
the TypeScript generation.
As a side note: I am not an expert with rules_js. This patch is my
attempt based on my limited understanding of the rule set.
Fixes#7817
* Fix the query
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Start using pnpm
* Add @npm
* get more stuff set up
* Get the analysis phase passing.
* Get esbuild working?
* Get it compiling?
$ bazel build //tests/ts/...
* Try to get the test working
* test is passing
* Get the other tests working
* clarify comment
* clean up a bit
* Try to add another test
* Add another test
* clean up more
* remove unused reference
* Add e2e test
* Get more of the test working
* add lock file
* Get test working on its own
* Get e2e test passing
* fix infinite recursion
* Add comments
* clean up some more
* clean up more again
* Source typescript version from package.json
* run buildifier
* lint
* Fix unset `extra_env`
* Incorporate feedback
* run buildifier
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
* optionally generate type prefixes and suffixes for python code
* fix codegen error when qualified name is empty
* WIP: Python typing
* more progress towards python typing
* Further iterate on Python generated code typing
* clang-format
* Regenerate code
* add documentation for Python type annotations option
* generate code with Python type annotations
* handle forward references
* clang-format
This change allows user to decode binary with given schema to JSON
representation when schema defines union with struct.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Fix C/C++ Create<Type>Direct with sorted vectors
If a struct has a key the vector has to be sorted. To sort the vector
you can't use "const".
* Changes due to code review
* Improve code readability
* Add generate of JSON schema to string to lib
* option indent_step is supported
* Remove unused variables
* Fix break in test
* Fix style to be consistent with rest of the code
* [TS] Fix reserved words as arguments (#6955)
* [TS] Fix generation of reserved words in object api (#7106)
* [TS] Fix generation of object api
* [TS] Fix MakeCamel -> ConvertCase
* [C#] Fix collision of field name and type name
* [TS] Add test for struct of struct of struct
* Update generated files
* Add missing files
* [TS] Fix query of null/undefined fields in object api
* Add .Net verfier
* Add some fuzz tests for .Net
* Remove additional files
* Fix .net test
* Changes due to PR
* Fix generated files
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
ToCamelCase(input, true) converts first char to upper case, but
ToCamelCase(input, false) keeps the case of the first char. We are
changing its behavior to force a lower case.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* TS/JS: Export object based classes on entry
Along with the non object ones, for consistency. This is a regression
introduced recently.
Before:
`export { UpdateSettingsRequest } from './worker/update-settings-request.js';`
Now:
`export { UpdateSettingsRequest, UpdateSettingsRequestT } from './worker/update-settings-request.js';`
* only export object based classes for structs
Enums are not elegible.
---------
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Fix C/C++ Create<Type>Direct with sorted vectors
If a struct has a key the vector has to be sorted. To sort the vector
you can't use "const".
* Changes due to code review
* Improve code readability
* Add generate of JSON schema to string to lib
* option indent_step is supported
* Remove unused variables
* Fix break in test
* Fix style to be consistent with rest of the code
* [TS] Fix reserved words as arguments (#6955)
* [TS] Fix generation of reserved words in object api (#7106)
* [TS] Fix generation of object api
* [TS] Fix MakeCamel -> ConvertCase
* [C#] Fix collision of field name and type name
* [TS] Add test for struct of struct of struct
* Update generated files
* Add missing files
* [TS] Fix query of null/undefined fields in object api
* Fix collision if field name is equal to table name and used as key in an array
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
