String default values parsed from .fbs schemas are un-escaped by the IDL
parser (e.g., \x22 becomes a raw " byte), but code generators embed these
raw values directly into generated source code string literals. This allows
specially crafted .fbs files to break out of string literals and inject
arbitrary code into generated C++, Rust, TypeScript, and Swift source.
Fix by adding EscapeCodeGenString() helper that re-escapes string content
before embedding, and applying it to all 7 affected injection points across
5 code generators (C++, Rust, TypeScript, Swift, FBS).
Resolves the TODO comments in idl_gen_cpp.cpp and idl_gen_rust.cpp.
* fix(flatbuffers): use manual impl Default for struct object types
* fix: handle bool and float zero literals in struct object Default impl
* fix: regenerate all test bindings with generate_code.py
* fix: data type check on swift build
* fix: test large array on struct and enum
When building a FlexBuffer using the Builder and adding data that exceeds the default buffer size (2048 bytes), in _newOffset() a larger buffer is created, but never used. This results in a RangeError.
Resolve by actually replacing the too small with the new larger buffer. Add a test that verifies this by adding multiple large strings to a vector.
There is a typo in the schema.md file in documentation. An empty 'required' section was added right before the 'required' item in the middle of the attributes list. It even appears in the table of content, which might confuse readers, making it look like following attributes might be required while they are not. You can notice the issue there: https://flatbuffers.dev/schema/#attributes.
I did check that `mkdocs serve -f mkdocs.yml` does produce the expected output (the same attributes list without that extra empty `required` section in the middle) with my changes.
* [C#] Add GetBytes methods for fixed arrays
I wanted to direct access to fixed array bytes. I made some changes to the idl generator to create GetBytes functions following the same naming conventions used for vectors of scalar types. There was not a 'Length' field present to bound the existing index accessor so I added that too.
+ Add generic GetBytes for fixed length arrays of scalar types
+ Implement conditional compilation for ENABLE_SPAN_T:
- ENABLE_SPAN_T: Returns `Span<T>` using `MemoryMarshal.Cast<byte, T>()` as needed.
- Else: Returns `ArraySegment<byte>?` for raw byte access
+ Added tests reusing arrays_test.fbs definitions
+ Added const int Length field to support existing index based accessors
* [C#] Sync generated code for after adding GetBytes methods for fixed arrays
---------
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
* Adds tests for Rust single file mode
All existing tests only compile Rust code using --rust-module-root-file.
* Adds standalone include tests for Rust
The imports for these tests have been moved to their own file, as the existing intergration_test.rs file hides compilation issues from code generation due to symbols brought into scope outside of the generated code (e.g. `extern crate alloc`).
* Declare alloc crate in every Rust namespace
When performing code generation within a single file, extern crate alloc needs to be delcared to bring alloc into scope within every inner namespace.
* Regenerates generated schemas
The write_vtable() function's comment claimed to "fill the WIP vtable
with zeros" but make_space() only reserves memory without initializing
it. When using custom allocators with non-zeroed buffers, unset vtable
field entries would contain garbage instead of zero (which indicates
"use default value").
This fix explicitly zeros the vtable memory after reserving space,
matching the C++ implementation's buf_.fill_big() behavior.
Added regression test using a garbage-filled allocator (0xAA) that
verifies vtable entries for unset fields are properly zeroed.
Fixes#8894
* Add --ts-undefined-for-optionals command line option
# Details
- Fixes#7656
- Added a new `--ts-undefined-for-optionals` command line option for `flatc`.
- If enabled, generated TypeScript code uses `undefined` for optional fields rather than `null`.
* Also add TS generated test files
* Run `sh scripts/clang-format-git.sh`
* also add tests/ts/lalala-options.ts to the repo
* move new tests to tests/ts/optional_values dir
* add tests/ts/optional_values/optional_values_generated.cjs to the repo
* reuse existing optional_scalars.fbs and add new test
* add comma
* sh scripts/clang-format-git.sh
* remove comma
* sh scripts/clang-format-git.sh
* trying things
* sh scripts/clang-format-git.sh
* done
* address feedback
* sh scripts/clang-format-git.sh
* run `sh scripts/clang-format-git.sh`
* remove uneeded `eslint-disable @typescript-eslint/no-namespace` line
---------
Co-authored-by: José Luis Millán <jmillan@aliax.net>
* fix for https://github.com/google/flatbuffers/issues/8759
__vector_as_array<T> calling ByteBuffer.ToArray<T> with the length in bytes by multiplying len with ByteBuffer.Sizeof<T> and FlatBuffersExampleTests extended to call GetVectorOfLongsArray/GetVectorOfDoublesArray which failed without the fix
* first try to repair build-dotnet-windows
* syntax error fixed
* Update solution creation command in build workflow
add --format sln to the dotnet new command, maybe it is currently creating a .slnx instead?
* Fix generate_code script path
* [Python] Make StartVector public
Make StartVector vector public since it is already being used in
generated code
* [Python] Improve vector creation for Python API
Makes Python API for vectors cleaner like Rust and Swift
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
* chore(idl): Check for case insensitive keywords
Most languages are not affected by this change. In PHP, some names such
as Bool cannot be used because it is a reserved keyword for to the bool
data type. The field `keywords_casing` in the configs enables checking
all characters in lowercase against every keyword. This should be safe
as flatbuffers does not allow non-ASCII characters in its grammar.
* chore: Fix formatting to follow google's coding style for enums
* chore: Extract convert case to lower when CaseInsensitive
---------
Co-authored-by: Justin Davis <jtdavis777@gmail.com>
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Remove invalid dependency on FLATBUFFERS_GENERATE_HEADERS_SCHEMAS
add_dependencies() is for targets.
CMake 4.2.0 fails because of this (it shouldn't crash though, but that's another topic). See https://gitlab.kitware.com/cmake/cmake/-/issues/27415
* Use FLATC_TARGET
---------
Co-authored-by: Justin Davis <jtdavis777@gmail.com>
* generate mutable union accessors
* add test
* Revert "add test"
This reverts commit 45e352b18f.
* update file
* formatter got in the way
* merge conflicts
* updated genned code
* manually fix code gen bc I can't figure out why this file won't code gen
---------
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
Add missing bounds checking to ByteVector before slice
operations in the Go FlatBuffers implementation. Relative offset and
vector length are now checked against the buffer size. Instead of
panicking, the code now returns nil. Regression test added.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
Co-authored-by: Justin Davis <jtdavis777@gmail.com>
* arrays of enums with no value for 0 now throw errors
* move setting key field outside struct check
* set to default instead of required
* unsure of why these bfbs files have changed at this time, checking them in to run the pipelines.
* remove known bad test
* flatc builds and seems to work, some of the extra targets are having linker errors
* fix build system
* pipeline failures
* un-rename files
* refactor to use unique_ptr
* typo
* rm make_unique, add comments
* fix cmake
---------
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
Improves unpack performance for vectors by allowing the compiler
to vectorize flatbuffers::Vector to std::vec::Vec conversions,
using the unstable trusted_len feature.
Internally, enables an optimization in src/alloc/vec/spec_extend.rs:
Previously, unpacking a flatbuffers::Vector called
SpecExtend::extend_desugared fallback, which inhibits vectorization
(due to a branch before every element move). Declaring TrustedLen
allows SpecExtend::extend_trusted, which LLVM can often vectorize
into a memcpy.
For [ubyte] vectors in particular, this turns a rather expensive
loop of 'mov BYTE PTR [rax+r13*1], bpl' into a `call memcpy`.
* add deser support for enum type
* update generated files
* remove deser generator when bitflag enable
* add deser test
* Restore the Rust editions version
* Remove unnecessary modifications
* Fixes checks for serde features in flexbuffers crate
* Removes unused MapReaderIndexer use statement
* Fixes warning about nightly cfg usage
Enabling a cfg attribute through cargo::rustc-cfg in build.rs should be coupled with a cargo::rust-check-cfg value so that the compiler knows about the custom cfg. See: https://doc.rust-lang.org/rustc/check-cfg/cargo-specifics.html#cargorustc-check-cfg-for-buildrsbuild-script.
* Migrates usage of deprecated float constants
This update fixes a compiler warning from use of the old constants.
Constants like EPSILON are now directly on the float primitives (e.g. f32::EPSILON) rather than in the f32 module (std::f32::EPSILON).
The new constants have existed since 1.43.0, which appears to be below the MSRV for the flatbuffers crate.
* Fixes incorrect key in flatbuffers Cargo.toml
The old code was using package.rust, which triggered a warning about an unused key:
warning: flatbuffers/rust/flatbuffers/Cargo.toml: unused manifest key: package.rust
The correct key for specifying MSRV is rust-version. See: https://doc.rust-lang.org/cargo/reference/rust-version.html#rust-version.
---------
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
* [Java] Use Table's fully qualified path
When a table's name is called `Table`, the Java bindings generated result in an error due to there being 2 Tables. This fixes the issue by fully qualifyng the flatbuffers Table import.
* Update codegen
* Update generated Java code
---------
Co-authored-by: Neville Dipale <neville@urbanlogiq.com>
* [Go] Write required string fields into the buffer when using Object API
In C++, CreateX allows to write the default "" value of a required
string, when the string is not explicitly set. Object API Pack method
uses this implementation of CreateX.
However, in go, despite whether the field is optional or required, it is
always checked against empty string allowing to create messages that
fail to pass the verifier. This commits partially reverts #7719 when the
string field is required.
* Add test for serializing required string fields using Object API
* Update generated code
The Monster schema contains a key string field. For historical
convenience reasons, string keys are assumed required.
* fix CScript string.compare
Because the default compare sorting rules of c# string are different from those of cpp, the binary file exported by flatc -b cannot use LookupByKey
* run generate_code.py
---------
Android libraries include <android/api-level.h> which defines the __ANDROID_API__ symbol even when targeting non-Android platforms and not using Android's libc. This updates the FLATBUFFERS_LOCALE_INDEPENDENT ifdef to check for __ANDROID__ before checking the Android API level.
Removes an extra check from the __Fuchsia__ branch. Fuchsia's libc does not support locales or the locale independent entry points.
Updates the Android API check to check for an API level >= 26 instead of 21. This matches the Android header file's availability macros and Bionic documentation:
https://android.googlesource.com/platform/bionic/+/HEAD/docs/status.md
This tried to generate from a directories "MyGame/Sample/"
for a empty path_ in M, MyGame & MyGame/Sample.
Which is incorrect since we want to start with the first
kPathSeparator `/` and not position 1.
We can't use macos-latest-large in non paid GitHub account but we can
use macos-15-intel in public repositories. If we use macos-15-intel,
we can run CI jobs for Intel macOS in fork repositories.
* Fixes to make SizeVerifier work.
In particular change all the places in the Flatbuffers library
and generated code that were using `Verifier` to instead use
`VerifierTemplate<TrackBufferSize>` and wrap them all inside
`template <bool TrackBufferSize = false>`.
Also add unit tests for SizeVerifier.
* Format using `sh scripts/clang-format-git.sh`
* Use `B` rather than `TrackBufferSize` for the name of the template parameter.
* Update generated files.
Implements InlineArrays which allow us to use Flatbuffers arrays within
Structs natively, and also implements FlatbufferVectors as a secondary API
when using mutable Structs
Fixes mutations within fixed sizes arrays
Adds tests and remove inout and mutating from generated objects in favor of borrowing
---------
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
* Support native_type for tables when using the C++ object API.
If native_type is specified on a table:
- No object API struct type is generated.
- The object API refers to the table by its native_type.
- UnPack and Create<TableName> methods are declared but not defined; as they
must be user-provided.
* Add tests for native_type on tables.
* Add documentation for native_type on tables.
Per the definition of --gen-compare: only generate comparators for object API
generated structs. Types annoated with native_type must define their own
comparators if `--gen-compare` is enabled.
Also enables --gen-compare for native_type_test and fixes the test by adding a
comparator for the Native::Vector3D type.
* Implements FlatbuffersVector in swift
Implements FlatbuffersVector which confirms to RandomAccessCollection,
this would give us semi-native sugary syntax to all the arrays in swift port.
This work will also be the foundation of using arrays in swift
* Fix failing tests for Swift
Previously: X::Pack forwarded to CreateX.
Now: CreateX will forward to X::Pack.
This is a step toward enabling using native types for tables when using the
object API. When defining a native table, the user will be able to define a
custom X::Pack method (which is more consistent with the existing native_type
functionality for structs). By reversing the order of the dependencies, CreateX
can continue to be auto-generated and will use the custom X::Pack method when
overriden for native_type tables.
* Prevent `make_span` from working with vectors and arrays of pointers
* support `make_structs_span` for little-endian
* fix build: add the required parentheses
---------
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
Treat flatbuffers' definition of npm_typescript as a dev dependency, in order to avoid conflicts when consuming flatbuffers in a repo that also depends on aspect_rules_ts.
* Correction of bug inside ToArray<T> methods
Avoid allocating too large buffers (len is expressed in bytes, not in Ts).
Added validation to ensure len is a multiple of SizeOf<T>() before converting to array.
* Update ByteBuffer.cs
* Refactor ToArray and ToArrayPadded methods
I understand from failed test that pos, len, padLeft and padRight are expressed in Ts
* Refactor ToArray and ToArrayPadded methods
* Final correction
All functions parameters expressed in bytes for homogeneity
Tests run:
- UNSAFE_BYTEBUFFER=true/ENABLE_SPAN_T=true: passed
- UNSAFE_BYTEBUFFER=true/ENABLE_SPAN_T=false: passed
- UNSAFE_BYTEBUFFER=false/ENABLE_SPAN_T=false: passed
- UNSAFE_BYTEBUFFER=false/ENABLE_SPAN_T=true: configuration forbidden by compilation
Correction of FlatBuffers.Test.csproj to allow UNSAFE_BYTEBUFFER/ENABLE_SPAN_T tests
Correction of FlatBuffersExampleTests.cs: I think the test was not run because it could not pass (to be reviewed carefully)
* Added ToSizedArrayPadded(int padLeft, int padRight) + ToArrayPadded(pos, len, padLeft, padRight) to the byteBuffers.
This is for API completion and to avoid unnecessary copy when framing my packets. I needed this to create a flat buffer with space in front of it for header / metadata.
* Fix indentation
---------
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
There are a couple instances where the ByteBuffer's Span property was accessed in a loop.
+ Extracted the use of the property outside of the loop to save a few cpu cycles.
Access to the allocator's internal buffer isn't exposed as a ReadOnlySpan<byte> from the ByteBuffer
or the FlatBufferBuilder.
+ Added a few convenience functions to access the buffer using a ReadOnlySpan<byte>.
There are a few cases where built in Span extensions can be used to run optimized code.
+ Added the use of Span.Fill() and ReadOnlySpan.SequenceCompareTo to replace existing loops.
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
new_p is a local addr but is owned now by slice_
thus the life time does not end at the end of the function
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
* fix(go/grpc): avoid panic on short FlatBuffers input
The gRPC codec read the root UOffsetT without checking input size. On
buffers shorter than SizeUOffsetT, GetUint32 touched data[3] and the
process panics.
Add a simple length check and validate the root offset stays within the
buffer. Return clear errors (insufficient data / invalid root offset)
instead of panicking.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
* fix(go/grpc): avoid signed overflow in offset
Keep the bounds check in the unsigned domain (UOffsetT) to avoid
signedness pitfalls.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
* chore: clarify comment regarding offset
A full FlatBuffer structure would be:
- uoffset_t: root table offset (4 bytes)
- soffset_t: vtable offset in root table (4 bytes)
- uint16_t: vtable size (2 bytes)
- uint16_t: table size (2 bytes)
In total 12 bytes. We are only validating the data length
before trying to read the uoffset_t, not the full structure.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
---------
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
Co-authored-by: Derek Bailey <derekbailey@google.com>
Our CI is broken and this is the error:
```
FAILURE: Build failed with an exception.
* What went wrong:
Gradle requires JVM 17 or later to run. Your build is currently configured to use JVM 11.
```
So updating our java-versions to the latest stable version which is 21 apparently.
Makes the return type of `static getFullyQualifiedName()` be a string literal instead of just the string type
Update tests
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
* [Python] Sync PythonTest.sh flags with generate_code.py
* [Python] Update generated code to latest flatc version for tests
* [Python] Fix test support for numpy newer than 2.0.0
* [Python] Remove unused variable
* [Python] Fix __eq__ for numpy arrays
* [Python] Run clang-format over the entire file
* Developers intro how to contribute
* Fix Rust code generation for Rust edition 2024
The errors look like:
```
warning[E0133]: call to unsafe function `fbs::flatbuffers::emplace_scalar` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::follow_cast_ref` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::Follow::follow` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::read_scalar_at` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::root_unchecked` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::size_prefixed_root_unchecked` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::Table::<'a>::new` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `std::slice::from_raw_parts` is unsafe and requires unsafe block
```
* Update goldens
Ran `goldens/generate_goldens.py`
* Regenerate code files
Ran `scripts/generate_code.py`
This commit significantly improves the developer experience for the Python Object-Based API by overhauling the generated `__init__` method for `T`-suffixed classes.
Previously, `T` objects had to be instantiated with an empty constructor, and their fields had to be populated manually one by one. This was verbose and not idiomatic Python.
This change modifies the Python code generator (`GenInitialize`) to produce `__init__` methods that are:
1. **Keyword-Argument-Friendly**: The constructor now accepts all table/struct fields as keyword arguments, allowing for concise, single-line object creation.
2. **Fully Typed**: The signature of the `__init__` method is now annotated with Python type hints. This provides immediate benefits for static analysis tools (like Mypy) and IDEs, enabling better autocompletion and type checking.
3. **Correctly Optional**: The generator now correctly wraps types in `Optional[...]` if their default value is `None`. This applies to strings, vectors, and other nullable fields, ensuring strict type safety.
The new approach remains **fully backward-compatible**, as all arguments have default values. Existing code that uses the empty constructor will continue to work without modification.
#### Example of a Generated `__init__`
**Before:**
```python
class KeyValueT(object):
def __init__(self):
self.key = None # type: str
self.value = None # type: str
```
**After:**
```python
class KeyValueT(object):
def __init__(self, key: Optional[str] = None, value: Optional[str] = None):
self.key = key
self.value = value
```
#### Example of User Code
**Before:**
```python
# Old, verbose way
kv = KeyValueT()
kv.key = "instrument"
kv.value = "EUR/USD"
```
**After:**
```python
# New, Pythonic way
kv = KeyValueT(key="instrument", value="EUR/USD")
```
Using the : syntax leads to non member attributes.
> If an attribute is defined in the class body with a type annotation
> but with no assigned value, a type checker should assume this is a non-member attribute
```
class Pet(Enum):
genus: str # Non-member attribute
species: str # Non-member attribute
CAT = 1 # Member attribute
DOG = 2 # Member attribute
```
https://typing.python.org/en/latest/spec/enums.html#defining-members
This prevents the include of the type defined in the pyi,
otherwise this leads to error message like this:
error: Name XYZ already defined (possibly by an import) [no-redef]
* [Python] Use correct type for str with None
Otherwise mypy will correctly flag code like this
def __init__(self):
self.fooBar = None # type: Optional[str]
error: Incompatible types in assignment (expression has type "None", variable has type "str")
* [Python] Make list type optional as they can contain None
Fixes an issue where exports were using incorrect relative paths for
>=3 namespace levels. This is fixed by making the starting range of the
namespace components relative to the amount of components.
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
Adding support for windows requires the code generations
to add a compiler statement to completely ignore GRPC code
generation on windows
Cleanup the project to use the main Package.swift to run tests
instead of having it separate and includes the imports for GRPC
within it.
Adds windows swift ci
* clang-format
* [Python] Replace . with _ in grpc filename suffix
Having filenames with . like `file.fb.grcp`
is not great for Python. Since dots are used for namespaces.
Replacing all of them with _ eg suffix `foo.bar.baz` will become
`foo_bar_baz`.
Restoring the previous default `_fb` suffix.
* [Python] Use namespace in path
This fixes a regression introduced with:
fb9afbafc7
And generates the grpc file in the namespace folder again.
* Sync commandline docs with web docs
* Offical Swift port for FlexBuffers
This is the offical port for FlexBuffers within
swift, and it introcudes a Common Module where code
is shared between flatbuffers and flexbuffers.
Writing most supported values like maps, vectors,
nil and scalars into a flexbuffer buffer. And includes
tests to verify that its similar to cpp
* Reading a flexbuffer
Implementing reading from a flexbuffer, enabling
most of the buffers features, like most types, maps, vectors,
typedvectors, and fixedtypedvectors.
Currently, if an offset/object cant be read we default to a swift
nil instead of the default flexbuffers 'null' with all values.
* Fixes bazel breaking due to new project structure
Address warnings within the library
* Adds comment on why we added the code & properly enforce the amout of bytes needed
For Vector or Array of structures the dereference operator of an
iterator returns the pointer to the structure. However, IndirectHelper,
which is used in the implementation of this operator, is instantiated
in the way that the IndirectHelper::Read returns structure by value.
This is because, Vector and Array instantiate IndirectHelper with
const T*, but VectorIterator instantiates IndirectHelper with T. There
are three IndirectHelper template definition: first for T, second for
Offset<T> and the last one for const T*. Those have different
IndirectHelper:Read implementations and (more importantly) return type.
This is the reason of mismatch in VectorIterator::operator* between
return type declaration and what was exactly returned.
That is, for Array<T,...> where T is scalar the VectorIterator is
instantiated as VectorIterator<T, T>, dereference operator returns T
and its implementation uses IndirectHelper<T> which Read function
returns T.
When T is not scalar, then VectorIterator is instantiated as
VectorIterator<T, const T *>, dereference operator returns const T * and
its implementation uses IndirectHelper<T> which Read function returns T.
The fix is done as follows:
* implement type trait is_specialization_of_Offset and
is_specialization_of_Offset64,
* change partial specialization of IndirectHelper with const T * that
it is instantiated by T and enabled only if T is not scalar and not
specialization of Offset or Offset64,
* remove type differentiation (due to scalar) from Array..
The above makes the IndirectHelper able to correctly instantiate itself
basing only on T. Thus, the instantiation in VectorIterator correctly
instantiate IndirectHelper::Read function, especially the return type.
Fixes access to union members when generating code with options "--cpp-field-case-style upper" and "--gen-object-api"
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
* Adds new API to reduce memory copying within swift
Adds new storage container _InternalByteBuffer which
will be holding the data that will be created within the swift
lib, however reading data will be redirected to ByteBuffer, which
should be able to handle all types of data that swift provide without
the need to copy the data itself. This is due to holding a reference to
the data.
Replaces assumingMemoryBinding with bindMemory which is safer
Adds function that provides access to a UnsafeBufferPointer for
scalars and NativeStructs within swift
Updates docs
Suppress compilation warnings by replacing var with let
Using overflow operators within swift to improve performance
Adds tests for GRPC message creation from a retained _InternalByteBuffer
* flatbuffers Rust reflection: replace num with num-traits
num crate is a wrapper over num-traits and a few other crates, that
reexports the APIs from all of them. We only need num-traits.
Signed-off-by: Marcin Radomski <dextero@google.com>
* Rust reflection: drop dependency on stdint crate
We only use it to get intmax_t for deriving alignment, which is an alias
for `core::ffi::c_long` [1]. We can use that directly instead.
[1] https://docs.rs/stdint/1.0.0/stdint/type.intmax_t.html
Signed-off-by: Marcin Radomski <dextero@google.com>
* Rust reflection: drop dependency on escape_string crate
It's used to format a string used for debugging only, so we might as
well use the builtin Debug representation of a string.
Signed-off-by: Marcin Radomski <dextero@google.com>
* Rust codegen: add derives on generated bitflags
Otherwise it limits the use of structs generated for reflection.fbs
in Rust reflection API.
Signed-off-by: Marcin Radomski <dextero@google.com>
* Rust flatbuffers: update bitflags dependency to 2.8
Signed-off-by: Marcin Radomski <dextero@google.com>
* Rust codegen: use bitflags v2 API for converting from bits
from_bits_unchecked was replaced with safe from_bits_retain.
Signed-off-by: Marcin Radomski <dextero@google.com>
* Regenerate Rust code after idl change
Signed-off-by: Marcin Radomski <dextero@google.com>
* Regenerate reflection_generated.rs
With flatc --rust ../../../reflection/reflection.fbs
Signed-off-by: Marcin Radomski <dextero@google.com>
* ts/BUILD.bazel: add missing import
Found by Buildifire presubmit:
Function "sh_binary" is not global anymore and needs to be loaded from
"@rules_shell//shell:sh_binary.bzl".
Signed-off-by: Marcin Radomski <dextero@google.com>
* Update expected value in generated_code_debug_prints_correctly test
In bitflags v2, the debug string representation of enum values is
different than it was in v1:
Blue -> Color(Blue)
(empty) -> LongEnum(0x0)
This change adjusts the expected test value.
Signed-off-by: Marcin Radomski <dextero@google.com>
* Fix tests build on Swift 5.8
grpc-swift 1.4.1 depends on swift-nio-ssl 2.14.0+ [1]. swift-nio-ssl 2.29.1
published on 2025-01-30, introduced some code [2] that uses a "switch
expression syntax" supported since Swift 5.9 [3]. Attempts to compile it with
Swift 5.8 cause build errors.
swift-nio-ssl project doesn't seem to support Swift 5.8. A commit from
2024-10-29 removes a "deprecated reference to a Swift 5.8 pipeline" [4].
swift-nio-ssl 2.29.0 is the last version that can be compiled with Swift
5.8. This commit pins it to that exact version.
[1] 66e27d7e84/Package.swift (L33)
[2] 3cb4d5ad12 (diff-bc1db1321ff689c2819245dcce1a3080554f0fc13f81b8d326c97e7d42717c8fR54)
[3] https://github.com/swiftlang/swift-evolution/blob/main/proposals/0380-if-switch-expressions.md
[4] 8a6b89d9a4
---------
Signed-off-by: Marcin Radomski <dextero@google.com>
Co-authored-by: Marcin Radomski <dextero@google.com>
This setup is much simpler than calling Bazel from within Bazel
and making sure files and flags are set up correctly.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Add missing file to filegroup for bazel integration tests
Fixup after a9257b6963.
* Align versions in bazel_respository_test_dir with root
* Update XCode version to 15.2
This is the oldest available version.
* Add WORKSPACE.bzlmod
* Add support for Bazel 7
* Add support for Bazel 8 in CI
* Restrict visibility of exported file
* Align npm_translate_lock attrs
* Remove defs_bzl_filename attr
* Align root_package with pnpm-lock.yaml location
Use a symlink to avoid copying the file.
* Update Buildkite Bazel CI
Restructure presubmit.yml to support matrix.
* Remove testing Ubuntu 18.04
The available LLVM 6.0 is too old to support std::filesystem.
* Add testing in Ubuntu 22.04
* Use Bazel version 6.5.0 in integration test
We need to copy the .fbs files into the package used for .bfbs files.
This is necessary as flatc doesn't provide support to specify the full
output file name for an .fbs file in a different folder.
I tried OUTPUT_FILE env var but this doesn't seem to be honored by
flatc.
Transitive headers like array.h have not been available in the runtime_cc target causing the build to fail. Adding all public headers to make sure transitive headers of flatbuffers.h are available.
* #Rust Create a crate for reflection
* #Rust Add a crate for reflection tests and helper to access schema
* #Rust Get root table of a buffer and access field with schema
* #Rust Add 'Struct' struct and corresponding getter
* #Rust Add functions of getting any table/struct field value as integer/float/string
* #Rust Add setters for scalar fields
* #Rust Add setter for string fields
* #Rust Add getter for Table/Vector fields
* #Rust Add buffer verification
* Add a 'SafeBuffer' struct which provides safe methods for reflection
It verifies buffer against schema during construction and provides all the unsafe getters in lib.rs in a safe way
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Improves vectors performance and adds a benchmark to vectors of offsets in swift
Improves performance for all arrays and for loops
Uses a tuple instead of allocating a struct each time we start iterating over fieldloc
Updates benchmark library
* Fixing swift Wasm ci
* Update grpc-core version
io.grpc:grpc-core package in version 1.36.0 contains multiple [CVE's](https://mvnrepository.com/artifact/io.grpc/grpc-core/1.36.0).
Bump grpc-core version to latest 1.68.0 version to mitigate potential vulnerabilities.
* Update grpc version to 1.67.1
grpc was mistakenly released to maven under version 1.68.0 whenever a real release was done for version 1.67.1 [1]. The mistake was fixed later.
[1] https://github.com/grpc/grpc-java/releases
Fix CVE-2022-25647
The package com.google.code.gson:gson before 2.8.9 is vulnerable to Deserialization of Untrusted Data via the writeReplace() method in internal classes, which may lead to denial of service attacks.
Bump up version of the gson package.
https://github.com/advisories/GHSA-4jrv-ppp4-jm57
This allows this type to meet the requirements of e.g.
std::ranges::range, which is necessary for it to work with the
std::span range constructor, or the "non-legacy" constructor for
Chromium's base::span.
Bug: none
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Update NativeObject.swift
Correct the word.
* Update ByteBuffer.swift
Type parameter does not existing, remove it.
* Update ByteBuffer.swift
Correct the word.
In case when flatbuffers are being used along with other project that defines "max" preprocessor macro, the ::max() in FLATBUFFERS_MAX_BUFFER_SIZE and FLATBUFFERS_MAX_64_BUFFER_SIZE is incorrectly being expanded to the macro. Adding parentheses enforces function-like interpretation.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Move `namer.h` and `idl_namer.h` to `include/codegen` so they can be reused from `grpc` dirqectory.
* [gRPC] Update the Python generator to produce typed handlers and Python stubs if requested.
* [gRPC] Document the newly added compiler flags.
This allows enums to be type check with mypy.
They will still behave like ints ->
> IntEnum is the same as Enum,
> but its members are also integers and can be used anywhere
> that an integer can be used.
> If any integer operation is performed with an IntEnum member,
> the resulting value loses its enumeration status.
https://docs.python.org/3/library/enum.html#enum.IntEnum
Only if the --python-typing flag is set.
* [Python] Generate `.pyi` stub files when `--python-typing` is on.
To support this change, the following modifications were made:
- added a new option to disable `numpy` helpers generation;
- added a new flag to control the target Python version:
`--python-version` can be one of the following:
- `0.x.x` – compatible with any Python version;
- `2.x.x` – compatible with Python 2;
- `3.x.x` – compatible with Python 3.
- added codegen utilities for Python;
- added a note that the generated .py file is empty.
* [C++] Update the validator to skip structs in namespaces other than the current one.
Updates copyright from 2023 to 2024 & formats code - updates formatting rules
Updates CI to run with swift 5.8
Adds wasmer & updates command to run carton as a swift plugin
Update bazelci to also accept swift 5.8
Adds swift 5.10 to the test matrix
* [Python] Generate `.pyi` stub files when `--python-typing` is on.
To support this change, the following modifications were made:
- added a new option to disable `numpy` helpers generation;
- added a new flag to control the target Python version:
`--python-version` can be one of the following:
- `0.x.x` – compatible with any Python version;
- `2.x.x` – compatible with Python 2;
- `3.x.x` – compatible with Python 3.
- added codegen utilities for Python;
- added a note that the generated .py file is empty.
* [Python] Update Bazel build rules.
* [Python] Update Bazel build rules.
* [Python] Run buildifier on BUILD.bazel files.
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
This will allow the code to be compiled with `-Wfloat-equal`
as this would result in the folowing warning/error:
vendor/flatbuffers/include/flatbuffers/base.h:465:69:
error: comparing floating point with == or != is unsafe [-Werror,-Wfloat-equal]
template<typename T> inline bool IsTheSameAs(T e, T def) { return e == def; }
But the way it is used in flatbuffers it is ok to compare floating
points with ==.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Fixes Bazel issues for windows and ci
Fetching boringssl within the flatbuffers repository, to patch the issues
of not being able to upgrade to Xcode 14.3 due to buildkite throwing
errors. The patch was inspired by the tenserflow patch
https://github.com/tensorflow/tensorflow/issues/60191#issuecomment-1496073147
Removes references of swift from the windows pipeline for bazel
Sets github actions to use xcode 14.3 for kotlin and sets the macOS
build for intel cpus.
* Update build.yml
Remove comment that is not relevant any longer.
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Update build.yml to ubuntu-24.04
Apparently g++-13 was removed from the ubuntu-22.04 runners.
We also don't have enterprise runners at 24.04 yet, so just use the free ones for now until we get support for those. CI builds might take longer now.
* Update build.yml
Downgrade to g++12 and revert change to using normal runners
* Update build.yml
Go back to ubuntu-24.04 and update both gcc and clang to their latest versions according to [this](https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2404-Readme.md?plain=1#L16-L20).
* Update build.yml
Go back to g++13 for now, as we get some exotic warning in g++14 for newer C++ standards.
* Update build.yml
Fix the other issues with `macos-latest` going to arm: https://github.com/actions/runner-images/tree/main?tab=readme-ov-file#available-images and that Swift wasn't installed in the ubuntu-24.04 by default.
* Update build.yml
Disable Kotlin MacOs CI
The new options are:
- `--grpc-filename-suffix` controls the suffix of the generated files;
- `--grpc-use-system-headers` controls the type of C++ includes generated;
- `--grpc-search-path` controls the directory that contains gRPC runtime;
- `--grpc-additional-header` allows to provide additional dependencies for the generated code.
* [Python] Fix various codegen problems.
This includes:
- escaping keywords happens **after** converting the case:
- currently, `table ClassT` generate `class = Class()` which is invalid Python;
- imports in `one_file` mode use the filename rather than the type name when resolving module names;
- use `filename_suffix` instead of the hardcoded `_generated` one;
- generate empty files if no structs or enums are available. This makes the set of output files more predictable for Bazel.
* [Python] Fix various codegen problems.
This includes:
- escaping keywords happens **after** converting the case:
- currently, `table ClassT` generate `class = Class()` which is invalid Python;
- imports in `one_file` mode use the filename rather than the type name when resolving module names;
- use `filename_suffix` instead of the hardcoded `_generated` one;
- generate empty files if no structs or enums are available. This makes the set of output files more predictable for Bazel.
When flatbuffers is being used from a project that has no use for
JavaScript, users encounter an error similar to the following:
ERROR: Skipping '@com_github_google_flatbuffers//:flatbuffers': error loading package '@com_github_google_flatbuffers//': Unable to find package for @npm//:defs.bzl: The repository '@npm' could not be resolved: Repository '@npm' is not defined.
WARNING: Target pattern parsing failed.
ERROR: error loading package '@com_github_google_flatbuffers//': Unable to find package for @npm//:defs.bzl: The repository '@npm' could not be resolved: Repository '@npm' is not defined.
INFO: Elapsed time: 0.023s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
currently loading: @com_github_google_flatbuffers//
That's not ideal. Users that only care about C++ for example
shouldn't be forced to deal with rules_js and friends.
This patch attempts to fix that by moving the rules_js-specific things
into the `ts` and `tests/ts` directories. This should allow
non-JavaScript projects to ignore rules_js and friends completely.
Here I basically followed the `rules_foo` example from rules_js:
https://github.com/aspect-build/rules_js/tree/main/e2e/rules_foo
The idea is that flatbuffers has its own npm dependencies regardless
of what other projects may have. This means we should not force the
user to import flatbuffers's npm dependencies. The new
`ts/repositories.bzl` file is used by dependents to import
flatbuffers's dependencies. They can still import their own
dependencies. This cleanup allowed me to move all
JavaScript-specific stuff from the top-level directory into
subdirectories.
There should be no changes in this patch in terms of functionality.
It's just a refactor of the rules_js call sites. Users will have to
add a call to the function in `ts/repositories.bzl` in their own
`WORKSPACE` file. They can use
`tests/ts/bazel_repository_test/WORKSPACE` as an example.
Co-authored-by: Derek Bailey <derekbailey@google.com>
If a schema contains a message named e.g. FooT and a message named Foo
while the Object API suffix is T, then two classes with colliding names
will be generated. This scenario will produce a C++ compiler error, but
it's confusing.
This patch moves the error to the compiler, allowing the user to more
readily act to correct the issue.
Co-authored-by: Michael Beardsworth <beardsworth@intrinsic.ai>
Fix "One Definition Rule" violation when using flatbuffers::Verifier with
FLATBUFFERS_TRACK_VERIFIER_BUFFER_SIZE defined in some compilation units
and not defined in other compilation units.
The fix is to make Verifier a template class, with a boolean template
parameter replacing the "#ifdef" conditionals; to rename it as
VerifierTemplate; and then to use "#ifdef" only for a "using" declaration
that defines the original name Verifier an an alias for the instantiated
template. In this way, even if FLATBUFFERS_TRACK_VERIFIER_BUFFER_SIZE is
defined in some compilation units and not in others, as long as clients
only reference flatbuffers::Verifier in .cc files, not header files, there
will be no ODR violation, since the only part whose definition varies is the
"using" declaration, which does not have external linkage.
There is still some possibility of clients creating ODR violations
if the client header files (rather than .cc files) reference
flatbuffers::Verifier. To avoid that, this change also deprecates
FLATBUFFERS_TRACK_VERIFIER_BUFFER_SIZE, and instead introduces
flatbuffers::SizeVerifier as a public name for the template instance with
the boolean parameter set to true, so that clients don't need to define
the macro at all.
* Reproduce the error in a unit test
Reproduces #8200
* Overload KeyCompareWithValue to work for string-like objects
This fixes#8200.
* Extra tests
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
It appears the upgrade to xcode 14.3 broke the macos build on builkite.
The last good build was using xcode 14.2, so go back to this version
until the issue is resolved.
When building with make, it was failing for me because the target grpctext doesn't exist. I strongly assume this was meant to be grpctest.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Add version of push which takes ContiguousBytes
* Ensure overloads aren't ambiguous
* Add version of createVector
* Add version of push which takes ContiguousBytes
* Ensure overloads aren't ambiguous
* Add version of createVector
* Add similar conditional to other use of ContiguousBytes
* Attempt CI fix
* Use memcpy instead of copyMemory
memcpy is faster in tests
* Add testContiguousBytes
* Add benchmarks
* Add version of createVector
* Add benchmarks
* Update push to copy memory
Since we don't care about endianness, we can simply memcpy the array of scalars
* Remove function and benchmarks
Since we don't care about endianness, a FixedWidthInteger version of createVector isn't needed
* Improve naming
* Add doc comment
* Fix C/C++ Create<Type>Direct with sorted vectors
If a struct has a key the vector has to be sorted. To sort the vector
you can't use "const".
* Changes due to code review
* Improve code readability
* Add generate of JSON schema to string to lib
* option indent_step is supported
* Remove unused variables
* Fix break in test
* Fix style to be consistent with rest of the code
* [TS] Fix reserved words as arguments (#6955)
* [TS] Fix generation of reserved words in object api (#7106)
* [TS] Fix generation of object api
* [TS] Fix MakeCamel -> ConvertCase
* [C#] Fix collision of field name and type name
* [TS] Add test for struct of struct of struct
* Update generated files
* Add missing files
* [TS] Fix query of null/undefined fields in object api
* Generate only files for comiled fbs (not for dependend ones)
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Update build.yml
Use our enterprise runners
* Make a default runs-on
* Update build.yml
Use the latest 64-core runners
* Update build.yml
Fix windows runner that don't have visual studios
* Update build.yml
use windows-2019 as the 2022 doesn't seem to have visual studios installed
* Update build.yml
Upgrade to gcc 13 and clang 15
* switch to __is_trivially_copyable
* fix cmake issue and warning about sign comparison
* Use libc++ for C++23 on clang for now
* Use libc++ for C++23 on clang for now
* exclude clang+15 for C++13 builds
Addresses a warning on xcode 15 regarding copying a pointer without safeguards
Address PR comments regarding initializing buffers with flag
Adds a test case for copying unaligned buffers
Formatting code
`buildifier` was complaining as follows:
#### :bazel: buildifier: found 2 lint issues in your WORKSPACE, BUILD and *.bzl files
<pre><code>tests/ts/bazel_repository_test_dir/BUILD:3:1: <a href="https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#out-of-order-load">out-of-order-load</a>: Load statement is out of its lexicographical order.
ts/BUILD.bazel:2:1: <a href="https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#out-of-order-load">out-of-order-load</a>: Load statement is out of its lexicographical order.</pre></code>
This can be fixed locally like so:
$ buildifier -lint fix $(git ls-files | grep -e '/BUILD.bazel$' -e '/BUILD$' -e '\<WORKSPACE$')
I also took this opportunity to fix one of the filenames.
I accidentally introduced these errors in #8078.
If you used flatbuffers_ts_library with gen_reflections = True then it
attempted to use the flat-file compiler rather than flatc itself.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* [Kotlin] Small optimizations and benchmark on deserialization
* [Kotlin] Remove redudant assign() method (use init() instead)
* [Kotlin] Fix benchmark run after change in flatbuffers-java deps
Commit 6e214c3a49 fixes Kotlin build,
but makes the kotlin-benchmark plugin misses the java classes at
runtime, causing NotClassFoundError. The alternative to solve the issue
is to read java's pom.xml to get the latest java version and use it
as dependency. With that we avoid compilation errors on a new version and
keep benchmark plugin happy.
By using specific jar version to use java's runtime on the benchmark
module we let CI break when new versions are released. So we are using
source directly instead
* [Kotlin] Introduction to Kotlin Multiplaform
The first implementation of the Kotlin code generation was made years
ago at the time Kotlin Multiplaform was not stable and Kotlin is mostly
used on JVM-based targets. For this reason the generated code uses java
based runtime.
That design decision comes with many drawbacks, leaving the code
generated more java-like and making it impossible to use more advanced
features of the Kotlin language.
In this change we are adding two parts: A pure, multi-plaform, Kotlin
runtime and a new code generator to accompany it.
* [Kotlin] Remove scalar sign cast from code generation
Now that we have a new runtime the accepts unsigned types, we don't
need to code generate casting back and from signed scalars. This
MR removes this from both code generations and adds the necessary
API to the runtime.
* [Kotlin] Use offset on public API to represent buffer position
Currently, kotlin was following Java's approach of representing objects,
vectors, tables as "Int" (the position of it in the buffer). This change
replaces naked Int with Offset<T>, offering a type-safe API. So,
instead of
fun Table.createTable(b: FlatBufferBuilder, subTable: Int)
We will have
fun Table.createTable(b: FlatBufferBuilder, subTable: Offset<SubTable>)
Making impossible to accidentally switch parameters.
The performance should be similar to use Int as we are using value
class for Offset and ArrayOffset, which most of the time translate to
Int in the bytecode.
* [Kotlin] Add builder for tables
Add builder constructor to make create of table more ergonomic.
For example the movie sample for the test set could be written as:
Movie.createMovie(fbb,
mainCharacterType = Character_.MuLan,
mainCharacter = att) {
charactersType = charsType
this.characters = characters
}
instead of:
Movie.startMovie(fbb)
Movie.addMainCharacterType(fbb, Character_.MuLan)
Movie.addMainCharacter(fbb, att as Offset<Any>)
Movie.addCharactersType(fbb, charsType)
Movie.addCharacters(fbb, charsVec)
Movie.endMovie(fbb)
* [Kotlin] Move enum types to value class
Moving to flatbuffer enums to value class adds type safety for parameters
with minimum to no performance impact.
* [Kotlin] Simplify Union parameters to avoid naked casting
Just a small change on the APIs that receive union as parameters,
creating a typealias UnionOffset to avoid using Offset<Any>. To "convert"
an table offset to an union, one just call Offset.toUnion().
* [Kotlin] Apply clang-format on kotlin code generators
* [Kotlin] Update kotlin generator to follow official naming conventions
Updating directory, package and enum naming to follow Kotlin official
convention.
https://kotlinlang.org/docs/coding-conventions.html#naming-rules
* [Kotlin] Add fixes to improve performance
1 - Add benchmark comparing serialization between Java & Kotlin
2 - ReadWriteBuffer does not auto-grow (thus avoid check size in every op)
3 - Add specialized add functions on FlatBufferBuilder to avoid boxing
offsets.
4 - Remove a few Kotlin syntax sugar that generated performance penalties.
* [Kotlin] Remove builder from Kotlin KMP and add some optimizations
to avoid boxing of Offset classes
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
The test was not actually invoking the bazel that was downloaded with
the `http_file` rule. I failed to add `executable = True` to the
`http_file` call. This caused the test to ignore that bazel binary and
went to the next one on the system.
This patch fixes the issue by adding the missing attribute. Also, this
patch changes the check in the test to make sure that the downloaded
file is indeed executable.
* First working hack of adding 64-bit. Don't judge :)
* Made vector_downward work on 64 bit types
* vector_downward uses size_t, added offset64 to reflection
* cleaned up adding offset64 in parser
* Add C++ testing skeleton for 64-bit
* working test for CreateVector64
* working >2 GiB buffers
* support for large strings
* simplified CreateString<> to just provide the offset type
* generalize CreateVector template
* update test_64.afb due to upstream format change
* Added Vector64 type, which is just an alias for vector ATM
* Switch to Offset64 for Vector64
* Update for reflection bfbs output change
* Starting to add support for vector64 type in C++
* made a generic CreateVector that can handle different offsets and vector types
* Support for 32-vector with 64-addressing
* Vector64 basic builder + tests working
* basic support for json vector64 support
* renamed fields in test_64bit.fbs to better reflect their use
* working C++ vector64 builder
* Apply --annotate-sparse-vector to 64-bit tests
* Enable Vector64 for --annotate-sparse-vectors
* Merged from upstream
* Add `near_string` field for testing 32-bit offsets alongside
* keep track of where the 32-bit and 64-bit regions are for flatbufferbuilder
* move template<> outside class body for GCC
* update run.sh to build and run tests
* basic assertion for adding 64-bit offset at the wrong time
* started to separate `FlatBufferBuilder` into two classes, 1 64-bit aware, the other not
* add test for nested flatbuffer vector64, fix bug in alignment of big vectors
* fixed CreateDirect method by iterating by Offset64 first
* internal refactoring of flatbufferbuilder
* block not supported languages in the parser from using 64-bit
* evolution tests for adding a vector64 field
* conformity tests for adding/removing offset64 attributes
* ensure test is for a big buffer
* add parser error tests for `offset64` and `vector64` attributes
* add missing static that GCC only complains about
* remove stdint-uintn.h header that gets automatically added
* move 64-bit CalculateOffset internal
* fixed return size of EndVector
* various fixes on windows
* add SizeT to vector_downward
* minimze range of size changes in vector and builder
* reworked how tracking if 64-offsets are added
* Add ReturnT to EndVector
* small cleanups
* remove need for second Array definition
* combine IndirectHelpers into one definition
* started support for vector of struct
* Support for 32/64-vectors of structs + Offset64
* small cleanups
* add verification for vector64
* add sized prefix for 64-bit buffers
* add fuzzer for 64-bit
* add example of adding many vectors using a wrapper table
* run the new -bfbs-gen-embed logic on the 64-bit tests
* remove run.sh and fix cmakelist issue
* fixed bazel rules
* fixed some PR comments
* add 64-bit tests to cmakelist
* Add binary schema reflection
* remove not-used parameter
* move logic from object API to base API
* forward declare
* remove duplicate code gen that was stompping on the edits
* reduce to just typedef generation
* fixed bazel rules to not stomp
* more bazel fixes to support additional generated files
* Migrate from rules_nodejs to rules_js/rules_ts (take 2)
This is the second version of patch #7923. The first version got
reverted because bazel query was failing:
$ bazel --nosystem_rc --nohome_rc query tests(set('//...')) except tests(attr("tags", "manual", set('//...')))
ERROR: Traceback (most recent call last):
File "/workdir/tests/ts/bazel_repository_test_dir/BUILD", line 6, column 22, in <toplevel>
npm_link_all_packages(name = "node_modules")
File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/external/npm/defs.bzl", line 188, column 13, in npm_link_all_packages
fail(msg)
Error in fail: The npm_link_all_packages() macro loaded from @npm//:defs.bzl and called in bazel package 'tests/ts/bazel_repository_test_dir' may only be called in bazel packages that correspond to the pnpm root package '' and pnpm workspace projects ''
This was happening because the `.bazelrc` file only added
`--deleted_packages` to the `build` command. We also need it for the
`query` command. This second version of the patch fixes that.
Original commit message:
This patch migrates the current use of rules_nodejs to the new rules_js.
rules_js is the intended replacement of rules_nodejs as per this note:
https://github.com/aspect-build/rules_js#relationship-to-rules_nodejs
> rules_js is an alternative to the build_bazel_rules_nodejs Bazel module
> and accompanying npm packages hosted in
> https://github.com/bazelbuild/rules_nodejs, which is now
> unmaintained. All users are recommended to use rules_js instead.
There are a few notable changes in this patch:
1. The `flatbuffer_ts_library` macro no longer accepts a `package_name`
attribute. This is because rules_js appears to manage the import
naming of dependencies via top-level `npm_link_package` targets.
Users will have to migrate.
2. I added a few more arguments to `flatbuffer_library_public()`. These
helped with exposing esbuild to `ts/compile_flat_file.sh`.
3. I pinned the version of `typescript` in `package.json` so that
rules_ts can download the exact same version. rules_ts doesn't know
what to do if the version isn't exact.
4. Since rules_js uses the pnpm locking mechanism, we now have a
`pnpm-lock.yaml` file instead of a yarn lock file.
4. I added bazel targets for a few of the existing tests in `tests/ts`.
They can be run with `bazel test //test/ts:all`. Since there is no
flexbuffers bazel target, I did not add a bazel target for the
corresponding test.
5. I added a separate workspace in `tests/ts/bazel_repository_test_dir/`
to validate that the flatbuffers code can be imported as an external
repository. You can run the test with
`bazel test //test/ts:bazel_repository_test`. For this to work, I
needed to expose a non-trivial chunk of the flatbuffers code to the
test. I achieved this through some recursive `distribution`
filegroups. This is inspired by rules_python's workspace tests.
I did not do anything special to validate that the `gen_reflections`
parameter works the same. This patch doesn't change anything about
the TypeScript generation.
As a side note: I am not an expert with rules_js. This patch is my
attempt based on my limited understanding of the rule set.
Fixes#7817
* Fix the query
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Start using pnpm
* Add @npm
* get more stuff set up
* Get the analysis phase passing.
* Get esbuild working?
* Get it compiling?
$ bazel build //tests/ts/...
* Try to get the test working
* test is passing
* Get the other tests working
* clarify comment
* clean up a bit
* Try to add another test
* Add another test
* clean up more
* remove unused reference
* Add e2e test
* Get more of the test working
* add lock file
* Get test working on its own
* Get e2e test passing
* fix infinite recursion
* Add comments
* clean up some more
* clean up more again
* Source typescript version from package.json
* run buildifier
* lint
* Fix unset `extra_env`
* Incorporate feedback
* run buildifier
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
absl::string_view is uses std::string_view when available. It already checks if std::string_view is available in the earlier code.
It should only use absl::string_view implementation.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* optionally generate type prefixes and suffixes for python code
* fix codegen error when qualified name is empty
* WIP: Python typing
* more progress towards python typing
* Further iterate on Python generated code typing
* clang-format
* Regenerate code
* add documentation for Python type annotations option
* generate code with Python type annotations
* handle forward references
* clang-format
This change allows user to decode binary with given schema to JSON
representation when schema defines union with struct.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Fix C/C++ Create<Type>Direct with sorted vectors
If a struct has a key the vector has to be sorted. To sort the vector
you can't use "const".
* Changes due to code review
* Improve code readability
* Add generate of JSON schema to string to lib
* option indent_step is supported
* Remove unused variables
* Fix break in test
* Fix style to be consistent with rest of the code
* [TS] Fix reserved words as arguments (#6955)
* [TS] Fix generation of reserved words in object api (#7106)
* [TS] Fix generation of object api
* [TS] Fix MakeCamel -> ConvertCase
* [C#] Fix collision of field name and type name
* [TS] Add test for struct of struct of struct
* Update generated files
* Add missing files
* [TS] Fix query of null/undefined fields in object api
* Add .Net verfier
* Add some fuzz tests for .Net
* Remove additional files
* Fix .net test
* Changes due to PR
* Fix generated files
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
PyPI has three capital letters. See the front page of the service:
https://pypi.org/
"The Python Package Index (PyPI) ..."
Update the Python link under "Supported programming languages"
Co-authored-by: Michael Le <michael.le647@gmail.com>
ToCamelCase(input, true) converts first char to upper case, but
ToCamelCase(input, false) keeps the case of the first char. We are
changing its behavior to force a lower case.
Co-authored-by: Derek Bailey <derekbailey@google.com>
Shorten the PR staleness from 6 months to 3 weeks + 1 week notice. PRs become much harder to deal with the old they become due to merge conflicts and divergence.
Updated to stale@v7.0.0
The distributions for C++ and Python include the generated reflection
bindings but are currently missing from the other language packages.
This will bring the Java package generated for releases closer to
feature parity with the C++ and Python release artifacts.
* TS/JS: Export object based classes on entry
Along with the non object ones, for consistency. This is a regression
introduced recently.
Before:
`export { UpdateSettingsRequest } from './worker/update-settings-request.js';`
Now:
`export { UpdateSettingsRequest, UpdateSettingsRequestT } from './worker/update-settings-request.js';`
* only export object based classes for structs
Enums are not elegible.
---------
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
Co-authored-by: Derek Bailey <derekbailey@google.com>
* [Android] fixed build after decomission of jcenter
JCenter[1] has been removed and now is failing android build. This
change updates the configuration to remove this and few other warnings.
1 - https://developer.android.com/studio/build/jcenter-migration
* [Kotlin] fix build for latest gradle version 8.0.1
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
The generation of the library interface supplied by this function only
works within the same directory as that the target was defined. By
adding a custom target named GENERATE_<TARGET> now also interface files
will be generated by making a target dependend on the generate target.
Example:
/CMakeLists.txt
set(MY_INCL_SRC ${CMAKE_CURRENT_SOURCE_DIR}/fbs/my_incl.fbs)
flatbuffers_generate_headers(TARGET my_incl
SCHEMAS ${MY_INCL_SRC})
add_subdirectory(app)
/app/CMakeLists.txt
add_executable(app src/test.cpp)
target_link_libraries(app my_incl)
add_dependencies(app GENERATE_my_incl)
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Fix C/C++ Create<Type>Direct with sorted vectors
If a struct has a key the vector has to be sorted. To sort the vector
you can't use "const".
* Changes due to code review
* Improve code readability
* Add generate of JSON schema to string to lib
* option indent_step is supported
* Remove unused variables
* Fix break in test
* Fix style to be consistent with rest of the code
* [TS] Fix reserved words as arguments (#6955)
* [TS] Fix generation of reserved words in object api (#7106)
* [TS] Fix generation of object api
* [TS] Fix MakeCamel -> ConvertCase
* [C#] Fix collision of field name and type name
* [TS] Add test for struct of struct of struct
* Update generated files
* Add missing files
* [TS] Fix query of null/undefined fields in object api
* Fix collision if field name is equal to table name and used as key in an array
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Add code generator for proto files
* Update
* Add --proto to script
* Remove cmt
* Move proto parsing logic into else block to share same set up logic for code_generator
* Remove IsValidCodeGenerator
If flatbuffers is built in C++11 mode, but there is a recent version of
absl which requires C++14, the build will fail.
Cf https://github.com/MapServer/MapServer/issues/6822 for the use case
that triggered this.
* Parsing from proto should keep field ID. (fixes#7645)
* Fix failed tests
* Fix windows warning
* Improve attribute generation in proto to fbs
* Check if id is used twice. fix Some clang-format problems
* Test if fake id can solve the test problem
* Validate proto file in proto -> fbs generation.
* Fix error messages
* Ignore id in union
* Add keep proto id for legacy and check gap flag have been added. Reserved id will be checked.
* Add needed flags
* unit tests
* fix fromat problem. fix comments and error messages.
* clear
* More unit tests
* Fix windows build
* Fix include problems
* Fake commit to invoke rebuild
* Fix buzel build
* Fix some issues
* Fix comments, fix return value and sort for android NDK
* Fix return type
* Break down big function
* Place todo
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
The current detection method fails on GCC 12.2 with -std=c++20 because
the __cpp_lib_span macro is undefined.
As per https://en.cppreference.com/w/cpp/utility/feature_test ,
__cpp_lib_span requires including either <version> or <span>.
Since both these headers were added in C++20, checking for C++20 is
sufficient (and simpler than using the library feature-test macro).
Signed-off-by: Bernie Innocenti <bernie@codewiz.org>
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Refactor to use CodeGenerator interface.
- Move code to its own header file to be included in flatc_main.cpp
- Refactor code to use CodeGenerator interface for all languages
* Format all files
* remove lua code generator since it doesn't support bfbs generator
* Update CMakeLists file with new idl_gen_*.cpp and idl_gen_*.h files
* Add idl_gen_swift header file
* Add idl_gen_swift header file and update bazel file
* Remove CodeGenerator interface for idl_gen_text.*. Remove comments and extern declaration
* Reorder header and implementation files in CMakeLists.txt
* Add idl_gen_* header files to implementation files
* Update CMakeLists and remove unused import
Co-authored-by: Derek Bailey <derekbailey@google.com>
* add unit tests for support struct as key
* make changes to parser and add helper function to generate comparator for struct
* implement
* add more unit tests
* format
* just a test
* test done
* rerun generator
* restore build file
* address comment
* format
* rebase
* rebase
* add more unit tests
* rerun generator
* address some comments
* address comment
* update
* format
* address comment
Co-authored-by: Wen Sun <sunwen@google.com>
Co-authored-by: Derek Bailey <derekbailey@google.com>
* [TS/JS] Entry point per namespace
* Fix handling of outputpath and array_test
* Attempt to fix generate_code
* Fix cwd for ts in generate_code
* Attempt to fixup bazel and some docs
* Add --ts-flat-files to bazel build to get bundle
* Move to DEFAULT_FLATC_TS_ARGS
* Attempt to add esbuild
* Attempt to use npm instead
* Remove futile attempt to add esbuild
* Attempt to as bazel esbuild
* Shuffle
* Upgrade bazel deps
* Revert failed attempts to get bazel working
* Ignore flatc tests for now
* Add esbuild dependency
* `package.json` Include esbuild
* `WORKSPACE` Add fetching esbuild binary
* Update WORKSPACE
* Unfreeze Lockfile
* Update WORKSPACE
* Update BUILD.bazel
* Rework to suggest instead of running external bundler
* Add esbuild generation to test script
* Prelim bundle test
* Run test JavaScriptTest from flatbuffers 1.x
* Deps upgrade
* Clang format fix
* Revert bazel changes
* Fix newline
* Generate with type declarations
* Handle "empty" root namespace
* Adjust tests for typescript_keywords.ts
* Separate test procedure for old node resolution module output
* Fix rel path for root level re-exports
* Bazel support for esbuild-based flatc
Unfortunately, we lose typing information because the new esbuild method
of generating single files does not generate type information.
The method used here is a bit hack-ish because it relies on parsing the
console output of flatc to figure out what to do.
* Try to fix bazel build for when node isn't present on host
* Auto formatting fixes
* Fix missing generated code
Co-authored-by: Derek Bailey <derekbailey@google.com>
Co-authored-by: James Kuszmaul <jabukuszmaul+collab@gmail.com>
* Fix binary output different in different platform, due to the nan serialization
* Add check generated code on windows ci
* Remove resdundant script
* Fix eof, and check script
* Minor bug in gen code script
* Fix windows script, remove redundant scripts
* Undelete redundante codes
* Fix github action
* Ignore eof generate grpc
Co-authored-by: Derek Bailey <derekbailey@google.com>
Comparing short strings, small integers, and Booleans by identity
(memory address) can work due to optimizations in the Python
interpreter, but it is neither formally correct nor reliable. Use
equality comparisons instead.
Co-authored-by: Derek Bailey <derekbailey@google.com>
It is deprecated in favour of importlib and slated for removal in Python
3.12. Since the return value of imp.find_module('numpy') is unused, the
only effect of calling this function is to raise an ImportError when
numpy is not available; importing numpy directly is already sufficient
to do this.
The imp package is still used in python/flatbuffers/compat.py, but only
on Python 2, where it is not deprecated and will not be removed.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* [TS]: Fix vtable creation for consecutive required fileds (#7739)
* handle feedback
* comment the schema
* comment change in builder.ts
* [TS]: builder, Fix requiredField()
Verifty that the field is present in the vtable.
* restore monsterdata binary file
Co-authored-by: Derek Bailey <derekbailey@google.com>
* [Kotlin] Control the generation of reflection with --reflect-names.
Tested:
```
$ cmake -G "Unix Makefiles" && make && ./tests/flatc/main.py
...
KotlinTests.EnumValAttributes
[PASSED]
KotlinTests.EnumValAttributes_ReflectNames
[PASSED]
KotlinTests: 2 of 2 passsed
...
35 of 35 tests passed
```
* [Kotlin] Fix SampleBinary by converting Byte to UByte for ubyte fields.
* [Kotlin] Annotate all generated classes with kotlin.ExperimentalUnsignedTypes.
Since flatbuffers is using calendar versioning and does not provide
any ABI stability guarantees, use the complete version as SOVERSION
for the shared library rather than just the major component. This
prevents breaking reverse dependencies on incompatible upgrades.
Fixes#7759
* Fix [C#] Object API - Invalid Property Name used in UnPackTo for union fieldhttps://github.com/google/flatbuffers/issues/7750, also fixes invalid Code generated in WriteJson for Unions named Value.
* Test added: new schema union_value_collision.fbs with a Union named Value and a union field named value. The generated C# code now compiles when NetTest.bat. The Code generated with an older flatc.exe didn't compile because of a mismatch of the property name (Value vs. Value_).
* branch was not up-to-date with master
* BASE_OPTS + CPP_OPTS removed and union_value_collision_generated.h deleted
Co-authored-by: Derek Bailey <derekbailey@google.com>
Add the --no-minmax-values flag to prevent flatc from generating C++
enums with MIN and MAX enumerated values that otherwise would be set
to the inclusive lower and upper bound respectively of the enum.
This command-line flag is needed to avoid collisions when an enum that
is being ported to FlatBuffers already has a MIN or MAX enumerated
value.
It is also needed to work around a long-standing problem with
magic_enum that causes magic_enum to not see enumerated values that
are not unique. For example, if FlatBuffers sets MIN = FOO and MAX =
BAR, MIN and FOO share the same underlying value so they are not
unique. The same is true of MAX and BAR. This prevents magic_enum
from converting FOO and BAR to and from strings as well as causing
magic_enum to return a count of enumerated values that is two fewer
than it should be.
Co-authored-by: Paul Serice <paul@serice.net>
* [Kotlin] Only generate nullable return types if the field is not required
* [Kotlin] Fix generated code formatting according to kotlin style guide
Co-authored-by: Derek Bailey <derekbailey@google.com>
Co-authored-by: Paulo Pinheiro <paulovictor.pinheiro@gmail.com>
To make it simple to map between a union field and its union type
field we are adding a pointer to FieldDef to point to each other. For
all other types the pointer will be nullptr.
Co-authored-by: Derek Bailey <derekbailey@google.com>
#7451 caused getFullyQualifiedName to return a name with underscores,
not periods. Because the fully qualified name is a property of
FlatBuffers, not the language being codegen'd for, it should use
periods. Fixes#7564.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Add clang-tidy, fix some bugpron problems.
* Fix more issues
* Fix some more issues :))
* Minimal pr to just add clang-tidy
Co-authored-by: Derek Bailey <derekbailey@google.com>
* create job to build linux and run unit test on s390x
* update
* update
* update
* update
* update
* print out machine type
* create regression test to build a big endian arch and run unit tests daily
* rename and schedule run on pr merged and on request
* udpate
Co-authored-by: Wen Sun <sunwen@google.com>
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Add Ref.AsStringBytes to flatbuffers.flexbuffers Python API
* Append Bytes to AsStringBytes return value
Co-authored-by: Jared Junyoung Lim <jaredlim@google.com>
Please delete this standard text once you've created your own description.
Please delete this standard text once you've created your own description.
If you make changes to any of the code generators (`src/idl_gen*`) be sure to
If you make changes to any of the code generators (`src/idl_gen*`) be sure to
build your project, as it will generate code based on the changes. If necessary
[build](https://google.github.io/flatbuffers/flatbuffers_guide_building.html) your project, as it will generate code based on the changes. If necessary
the code generation script can be directly run (`scripts/generate_code.py`),
the code generation script can be directly run (`scripts/generate_code.py`),
requires Python3. This allows us to better see the effect of the PR.
requires Python3. This allows us to better see the effect of the PR.
