* Add --ts-undefined-for-optionals command line option
# Details
- Fixes#7656
- Added a new `--ts-undefined-for-optionals` command line option for `flatc`.
- If enabled, generated TypeScript code uses `undefined` for optional fields rather than `null`.
* Also add TS generated test files
* Run `sh scripts/clang-format-git.sh`
* also add tests/ts/lalala-options.ts to the repo
* move new tests to tests/ts/optional_values dir
* add tests/ts/optional_values/optional_values_generated.cjs to the repo
* reuse existing optional_scalars.fbs and add new test
* add comma
* sh scripts/clang-format-git.sh
* remove comma
* sh scripts/clang-format-git.sh
* trying things
* sh scripts/clang-format-git.sh
* done
* address feedback
* sh scripts/clang-format-git.sh
* run `sh scripts/clang-format-git.sh`
* remove uneeded `eslint-disable @typescript-eslint/no-namespace` line
---------
Co-authored-by: José Luis Millán <jmillan@aliax.net>
* flatc builds and seems to work, some of the extra targets are having linker errors
* fix build system
* pipeline failures
* un-rename files
* refactor to use unique_ptr
* typo
* rm make_unique, add comments
* fix cmake
---------
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
* Move `namer.h` and `idl_namer.h` to `include/codegen` so they can be reused from `grpc` dirqectory.
* [gRPC] Update the Python generator to produce typed handlers and Python stubs if requested.
* [gRPC] Document the newly added compiler flags.
* [Python] Generate `.pyi` stub files when `--python-typing` is on.
To support this change, the following modifications were made:
- added a new option to disable `numpy` helpers generation;
- added a new flag to control the target Python version:
`--python-version` can be one of the following:
- `0.x.x` – compatible with any Python version;
- `2.x.x` – compatible with Python 2;
- `3.x.x` – compatible with Python 3.
- added codegen utilities for Python;
- added a note that the generated .py file is empty.
* [Python] Update Bazel build rules.
* [Python] Update Bazel build rules.
* [Python] Run buildifier on BUILD.bazel files.
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
The new options are:
- `--grpc-filename-suffix` controls the suffix of the generated files;
- `--grpc-use-system-headers` controls the type of C++ includes generated;
- `--grpc-search-path` controls the directory that contains gRPC runtime;
- `--grpc-additional-header` allows to provide additional dependencies for the generated code.
* [Kotlin] Introduction to Kotlin Multiplaform
The first implementation of the Kotlin code generation was made years
ago at the time Kotlin Multiplaform was not stable and Kotlin is mostly
used on JVM-based targets. For this reason the generated code uses java
based runtime.
That design decision comes with many drawbacks, leaving the code
generated more java-like and making it impossible to use more advanced
features of the Kotlin language.
In this change we are adding two parts: A pure, multi-plaform, Kotlin
runtime and a new code generator to accompany it.
* [Kotlin] Remove scalar sign cast from code generation
Now that we have a new runtime the accepts unsigned types, we don't
need to code generate casting back and from signed scalars. This
MR removes this from both code generations and adds the necessary
API to the runtime.
* [Kotlin] Use offset on public API to represent buffer position
Currently, kotlin was following Java's approach of representing objects,
vectors, tables as "Int" (the position of it in the buffer). This change
replaces naked Int with Offset<T>, offering a type-safe API. So,
instead of
fun Table.createTable(b: FlatBufferBuilder, subTable: Int)
We will have
fun Table.createTable(b: FlatBufferBuilder, subTable: Offset<SubTable>)
Making impossible to accidentally switch parameters.
The performance should be similar to use Int as we are using value
class for Offset and ArrayOffset, which most of the time translate to
Int in the bytecode.
* [Kotlin] Add builder for tables
Add builder constructor to make create of table more ergonomic.
For example the movie sample for the test set could be written as:
Movie.createMovie(fbb,
mainCharacterType = Character_.MuLan,
mainCharacter = att) {
charactersType = charsType
this.characters = characters
}
instead of:
Movie.startMovie(fbb)
Movie.addMainCharacterType(fbb, Character_.MuLan)
Movie.addMainCharacter(fbb, att as Offset<Any>)
Movie.addCharactersType(fbb, charsType)
Movie.addCharacters(fbb, charsVec)
Movie.endMovie(fbb)
* [Kotlin] Move enum types to value class
Moving to flatbuffer enums to value class adds type safety for parameters
with minimum to no performance impact.
* [Kotlin] Simplify Union parameters to avoid naked casting
Just a small change on the APIs that receive union as parameters,
creating a typealias UnionOffset to avoid using Offset<Any>. To "convert"
an table offset to an union, one just call Offset.toUnion().
* [Kotlin] Apply clang-format on kotlin code generators
* [Kotlin] Update kotlin generator to follow official naming conventions
Updating directory, package and enum naming to follow Kotlin official
convention.
https://kotlinlang.org/docs/coding-conventions.html#naming-rules
* [Kotlin] Add fixes to improve performance
1 - Add benchmark comparing serialization between Java & Kotlin
2 - ReadWriteBuffer does not auto-grow (thus avoid check size in every op)
3 - Add specialized add functions on FlatBufferBuilder to avoid boxing
offsets.
4 - Remove a few Kotlin syntax sugar that generated performance penalties.
* [Kotlin] Remove builder from Kotlin KMP and add some optimizations
to avoid boxing of Offset classes
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
* First working hack of adding 64-bit. Don't judge :)
* Made vector_downward work on 64 bit types
* vector_downward uses size_t, added offset64 to reflection
* cleaned up adding offset64 in parser
* Add C++ testing skeleton for 64-bit
* working test for CreateVector64
* working >2 GiB buffers
* support for large strings
* simplified CreateString<> to just provide the offset type
* generalize CreateVector template
* update test_64.afb due to upstream format change
* Added Vector64 type, which is just an alias for vector ATM
* Switch to Offset64 for Vector64
* Update for reflection bfbs output change
* Starting to add support for vector64 type in C++
* made a generic CreateVector that can handle different offsets and vector types
* Support for 32-vector with 64-addressing
* Vector64 basic builder + tests working
* basic support for json vector64 support
* renamed fields in test_64bit.fbs to better reflect their use
* working C++ vector64 builder
* Apply --annotate-sparse-vector to 64-bit tests
* Enable Vector64 for --annotate-sparse-vectors
* Merged from upstream
* Add `near_string` field for testing 32-bit offsets alongside
* keep track of where the 32-bit and 64-bit regions are for flatbufferbuilder
* move template<> outside class body for GCC
* update run.sh to build and run tests
* basic assertion for adding 64-bit offset at the wrong time
* started to separate `FlatBufferBuilder` into two classes, 1 64-bit aware, the other not
* add test for nested flatbuffer vector64, fix bug in alignment of big vectors
* fixed CreateDirect method by iterating by Offset64 first
* internal refactoring of flatbufferbuilder
* block not supported languages in the parser from using 64-bit
* evolution tests for adding a vector64 field
* conformity tests for adding/removing offset64 attributes
* ensure test is for a big buffer
* add parser error tests for `offset64` and `vector64` attributes
* add missing static that GCC only complains about
* remove stdint-uintn.h header that gets automatically added
* move 64-bit CalculateOffset internal
* fixed return size of EndVector
* various fixes on windows
* add SizeT to vector_downward
* minimze range of size changes in vector and builder
* reworked how tracking if 64-offsets are added
* Add ReturnT to EndVector
* small cleanups
* remove need for second Array definition
* combine IndirectHelpers into one definition
* started support for vector of struct
* Support for 32/64-vectors of structs + Offset64
* small cleanups
* add verification for vector64
* add sized prefix for 64-bit buffers
* add fuzzer for 64-bit
* add example of adding many vectors using a wrapper table
* run the new -bfbs-gen-embed logic on the 64-bit tests
* remove run.sh and fix cmakelist issue
* fixed bazel rules
* fixed some PR comments
* add 64-bit tests to cmakelist
* optionally generate type prefixes and suffixes for python code
* fix codegen error when qualified name is empty
* WIP: Python typing
* more progress towards python typing
* Further iterate on Python generated code typing
* clang-format
* Regenerate code
* add documentation for Python type annotations option
* generate code with Python type annotations
* handle forward references
* clang-format
* Add code generator for proto files
* Update
* Add --proto to script
* Remove cmt
* Move proto parsing logic into else block to share same set up logic for code_generator
* Remove IsValidCodeGenerator
* Parsing from proto should keep field ID. (fixes#7645)
* Fix failed tests
* Fix windows warning
* Improve attribute generation in proto to fbs
* Check if id is used twice. fix Some clang-format problems
* Test if fake id can solve the test problem
* Validate proto file in proto -> fbs generation.
* Fix error messages
* Ignore id in union
* Add keep proto id for legacy and check gap flag have been added. Reserved id will be checked.
* Add needed flags
* unit tests
* fix fromat problem. fix comments and error messages.
* clear
* More unit tests
* Fix windows build
* Fix include problems
* Fake commit to invoke rebuild
* Fix buzel build
* Fix some issues
* Fix comments, fix return value and sort for android NDK
* Fix return type
* Break down big function
* Place todo
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
* [TS/JS] Entry point per namespace
* Fix handling of outputpath and array_test
* Attempt to fix generate_code
* Fix cwd for ts in generate_code
* Attempt to fixup bazel and some docs
* Add --ts-flat-files to bazel build to get bundle
* Move to DEFAULT_FLATC_TS_ARGS
* Attempt to add esbuild
* Attempt to use npm instead
* Remove futile attempt to add esbuild
* Attempt to as bazel esbuild
* Shuffle
* Upgrade bazel deps
* Revert failed attempts to get bazel working
* Ignore flatc tests for now
* Add esbuild dependency
* `package.json` Include esbuild
* `WORKSPACE` Add fetching esbuild binary
* Update WORKSPACE
* Unfreeze Lockfile
* Update WORKSPACE
* Update BUILD.bazel
* Rework to suggest instead of running external bundler
* Add esbuild generation to test script
* Prelim bundle test
* Run test JavaScriptTest from flatbuffers 1.x
* Deps upgrade
* Clang format fix
* Revert bazel changes
* Fix newline
* Generate with type declarations
* Handle "empty" root namespace
* Adjust tests for typescript_keywords.ts
* Separate test procedure for old node resolution module output
* Fix rel path for root level re-exports
* Bazel support for esbuild-based flatc
Unfortunately, we lose typing information because the new esbuild method
of generating single files does not generate type information.
The method used here is a bit hack-ish because it relies on parsing the
console output of flatc to figure out what to do.
* Try to fix bazel build for when node isn't present on host
* Auto formatting fixes
* Fix missing generated code
Co-authored-by: Derek Bailey <derekbailey@google.com>
Co-authored-by: James Kuszmaul <jabukuszmaul+collab@gmail.com>
Add the --no-minmax-values flag to prevent flatc from generating C++
enums with MIN and MAX enumerated values that otherwise would be set
to the inclusive lower and upper bound respectively of the enum.
This command-line flag is needed to avoid collisions when an enum that
is being ported to FlatBuffers already has a MIN or MAX enumerated
value.
It is also needed to work around a long-standing problem with
magic_enum that causes magic_enum to not see enumerated values that
are not unique. For example, if FlatBuffers sets MIN = FOO and MAX =
BAR, MIN and FOO share the same underlying value so they are not
unique. The same is true of MAX and BAR. This prevents magic_enum
from converting FOO and BAR to and from strings as well as causing
magic_enum to return a count of enumerated values that is two fewer
than it should be.
Co-authored-by: Paul Serice <paul@serice.net>
To make it simple to map between a union field and its union type
field we are adding a pointer to FieldDef to point to each other. For
all other types the pointer will be nullptr.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Add --go-module-name flag to support generating code for go modules
* Rename echo example folder
* Grammar
* Update readme for go-echo example
* Update readme for go-echo example
* Re-enable go modules after test is done
The BytesConsumed function uses the `cursor_` to determine how many
bytes have been consumed by the parser, in case the user of the Parser
object wants to step over the parsed flatbuffer that is embedded in some
larger string. However, the `cursor_` is always one token ahead, so that
it can determine how to consume it. It points at the token that is about
to be consumed, which is ahead of the last byte consumed.
For example, if you had a string containing these two json objects and
parsed them...
"{\"key\":\"value\"},{\"key\":\"value\"}"
...then the `cursor_` would be pointing at the comma between the two
tables. If you were to hold a pointer to the beginning of the string and
add `BytesConsumed()` to it like so:
const char* json = // ...
parser.ParseJson(json);
json += parser.BytesConsumed();
then the pointer would skip over the comma, which is not the expected
behavior. It should only consume the table itself.
The solution is simple: Just hold onto a previous cursor location and
use that for the `BytesConsumed()` call. The previous cursor location
just needs to be set to the cursor_ location each time the cursor_ is
about to be updated. This will result in `BytesConsumed()` returning
the correct number of bytes without the off-by-one-token error.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Add support for proto 3 map to fbs gen
* Run clang-format
* Update proto golden test
* Rename variables
* Remove iostream
* Remove iostream
* Run clang format
Co-authored-by: Derek Bailey <derekbailey@google.com>
Previously when parsing a JSON representation of a Flatbuffer, the
parser required that the input string contain one and only one root
table. This change adds a flag that removes that requirement, so that
if a Flatbuffer table is embedded in some larger string the parser will
simply stop parsing once it reaches the end of the root table, and does
not validate that it has reached the end of the string.
This change also adds a BytesConsumed function, which returns the number
of bytes the parser consumed. This is useful if the table embedded in
some larger string that is being parsed, and that outer parser needs to
know how many bytes the table was so that it can step over it.
* Bfbs Nim Generator
* Remove commented out tests
* add missing line to idl.h
* Commit python reflection changes
* Commit python reflection changes and move tests
* Remove default string addition
* Move tests to python file
* Fix element size check when element is table
* remove whitespace changes
* add element_type docs and commit further to namer and remove kkeep
* Bfbs Nim Generator
* Remove commented out tests
* add missing line to idl.h
* Commit python reflection changes
* Commit python reflection changes and move tests
* Remove default string addition
* Move tests to python file
* Fix element size check when element is table
* remove whitespace changes
* add element_type docs and commit further to namer and remove kkeep
* remove unused variables
* added tests to ci
* added tests to ci
* fixes
* Added reflection type Field, Variable to namer
* Moved reflection namer impl to bfbsnamer
* Remove whitespace at end of line
* Added nim to generated code
* Revert whitespace removal
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Add support for metadata attributes for enum values (#7567)
* Fix path lookup in flatc test
* Try a fix for Windows paths
* Convert path to string to fix Windows error
