String default values parsed from .fbs schemas are un-escaped by the IDL
parser (e.g., \x22 becomes a raw " byte), but code generators embed these
raw values directly into generated source code string literals. This allows
specially crafted .fbs files to break out of string literals and inject
arbitrary code into generated C++, Rust, TypeScript, and Swift source.
Fix by adding EscapeCodeGenString() helper that re-escapes string content
before embedding, and applying it to all 7 affected injection points across
5 code generators (C++, Rust, TypeScript, Swift, FBS).
Resolves the TODO comments in idl_gen_cpp.cpp and idl_gen_rust.cpp.
* fix(flatbuffers): use manual impl Default for struct object types
* fix: handle bool and float zero literals in struct object Default impl
* fix: regenerate all test bindings with generate_code.py
* fix: data type check on swift build
* fix: test large array on struct and enum
When building a FlexBuffer using the Builder and adding data that exceeds the default buffer size (2048 bytes), in _newOffset() a larger buffer is created, but never used. This results in a RangeError.
Resolve by actually replacing the too small with the new larger buffer. Add a test that verifies this by adding multiple large strings to a vector.
There is a typo in the schema.md file in documentation. An empty 'required' section was added right before the 'required' item in the middle of the attributes list. It even appears in the table of content, which might confuse readers, making it look like following attributes might be required while they are not. You can notice the issue there: https://flatbuffers.dev/schema/#attributes.
I did check that `mkdocs serve -f mkdocs.yml` does produce the expected output (the same attributes list without that extra empty `required` section in the middle) with my changes.
* [C#] Add GetBytes methods for fixed arrays
I wanted to direct access to fixed array bytes. I made some changes to the idl generator to create GetBytes functions following the same naming conventions used for vectors of scalar types. There was not a 'Length' field present to bound the existing index accessor so I added that too.
+ Add generic GetBytes for fixed length arrays of scalar types
+ Implement conditional compilation for ENABLE_SPAN_T:
- ENABLE_SPAN_T: Returns `Span<T>` using `MemoryMarshal.Cast<byte, T>()` as needed.
- Else: Returns `ArraySegment<byte>?` for raw byte access
+ Added tests reusing arrays_test.fbs definitions
+ Added const int Length field to support existing index based accessors
* [C#] Sync generated code for after adding GetBytes methods for fixed arrays
---------
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
* Adds tests for Rust single file mode
All existing tests only compile Rust code using --rust-module-root-file.
* Adds standalone include tests for Rust
The imports for these tests have been moved to their own file, as the existing intergration_test.rs file hides compilation issues from code generation due to symbols brought into scope outside of the generated code (e.g. `extern crate alloc`).
* Declare alloc crate in every Rust namespace
When performing code generation within a single file, extern crate alloc needs to be delcared to bring alloc into scope within every inner namespace.
* Regenerates generated schemas
The write_vtable() function's comment claimed to "fill the WIP vtable
with zeros" but make_space() only reserves memory without initializing
it. When using custom allocators with non-zeroed buffers, unset vtable
field entries would contain garbage instead of zero (which indicates
"use default value").
This fix explicitly zeros the vtable memory after reserving space,
matching the C++ implementation's buf_.fill_big() behavior.
Added regression test using a garbage-filled allocator (0xAA) that
verifies vtable entries for unset fields are properly zeroed.
Fixes#8894
* Add --ts-undefined-for-optionals command line option
# Details
- Fixes#7656
- Added a new `--ts-undefined-for-optionals` command line option for `flatc`.
- If enabled, generated TypeScript code uses `undefined` for optional fields rather than `null`.
* Also add TS generated test files
* Run `sh scripts/clang-format-git.sh`
* also add tests/ts/lalala-options.ts to the repo
* move new tests to tests/ts/optional_values dir
* add tests/ts/optional_values/optional_values_generated.cjs to the repo
* reuse existing optional_scalars.fbs and add new test
* add comma
* sh scripts/clang-format-git.sh
* remove comma
* sh scripts/clang-format-git.sh
* trying things
* sh scripts/clang-format-git.sh
* done
* address feedback
* sh scripts/clang-format-git.sh
* run `sh scripts/clang-format-git.sh`
* remove uneeded `eslint-disable @typescript-eslint/no-namespace` line
---------
Co-authored-by: José Luis Millán <jmillan@aliax.net>
* fix for https://github.com/google/flatbuffers/issues/8759
__vector_as_array<T> calling ByteBuffer.ToArray<T> with the length in bytes by multiplying len with ByteBuffer.Sizeof<T> and FlatBuffersExampleTests extended to call GetVectorOfLongsArray/GetVectorOfDoublesArray which failed without the fix
* first try to repair build-dotnet-windows
* syntax error fixed
* Update solution creation command in build workflow
add --format sln to the dotnet new command, maybe it is currently creating a .slnx instead?
* Fix generate_code script path
* [Python] Make StartVector public
Make StartVector vector public since it is already being used in
generated code
* [Python] Improve vector creation for Python API
Makes Python API for vectors cleaner like Rust and Swift
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
* chore(idl): Check for case insensitive keywords
Most languages are not affected by this change. In PHP, some names such
as Bool cannot be used because it is a reserved keyword for to the bool
data type. The field `keywords_casing` in the configs enables checking
all characters in lowercase against every keyword. This should be safe
as flatbuffers does not allow non-ASCII characters in its grammar.
* chore: Fix formatting to follow google's coding style for enums
* chore: Extract convert case to lower when CaseInsensitive
---------
Co-authored-by: Justin Davis <jtdavis777@gmail.com>
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Remove invalid dependency on FLATBUFFERS_GENERATE_HEADERS_SCHEMAS
add_dependencies() is for targets.
CMake 4.2.0 fails because of this (it shouldn't crash though, but that's another topic). See https://gitlab.kitware.com/cmake/cmake/-/issues/27415
* Use FLATC_TARGET
---------
Co-authored-by: Justin Davis <jtdavis777@gmail.com>
* generate mutable union accessors
* add test
* Revert "add test"
This reverts commit 45e352b18f.
* update file
* formatter got in the way
* merge conflicts
* updated genned code
* manually fix code gen bc I can't figure out why this file won't code gen
---------
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
Add missing bounds checking to ByteVector before slice
operations in the Go FlatBuffers implementation. Relative offset and
vector length are now checked against the buffer size. Instead of
panicking, the code now returns nil. Regression test added.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
Co-authored-by: Justin Davis <jtdavis777@gmail.com>
* arrays of enums with no value for 0 now throw errors
* move setting key field outside struct check
* set to default instead of required
* unsure of why these bfbs files have changed at this time, checking them in to run the pipelines.
* remove known bad test
* flatc builds and seems to work, some of the extra targets are having linker errors
* fix build system
* pipeline failures
* un-rename files
* refactor to use unique_ptr
* typo
* rm make_unique, add comments
* fix cmake
---------
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
Improves unpack performance for vectors by allowing the compiler
to vectorize flatbuffers::Vector to std::vec::Vec conversions,
using the unstable trusted_len feature.
Internally, enables an optimization in src/alloc/vec/spec_extend.rs:
Previously, unpacking a flatbuffers::Vector called
SpecExtend::extend_desugared fallback, which inhibits vectorization
(due to a branch before every element move). Declaring TrustedLen
allows SpecExtend::extend_trusted, which LLVM can often vectorize
into a memcpy.
For [ubyte] vectors in particular, this turns a rather expensive
loop of 'mov BYTE PTR [rax+r13*1], bpl' into a `call memcpy`.
* add deser support for enum type
* update generated files
* remove deser generator when bitflag enable
* add deser test
* Restore the Rust editions version
* Remove unnecessary modifications
* Fixes checks for serde features in flexbuffers crate
* Removes unused MapReaderIndexer use statement
* Fixes warning about nightly cfg usage
Enabling a cfg attribute through cargo::rustc-cfg in build.rs should be coupled with a cargo::rust-check-cfg value so that the compiler knows about the custom cfg. See: https://doc.rust-lang.org/rustc/check-cfg/cargo-specifics.html#cargorustc-check-cfg-for-buildrsbuild-script.
* Migrates usage of deprecated float constants
This update fixes a compiler warning from use of the old constants.
Constants like EPSILON are now directly on the float primitives (e.g. f32::EPSILON) rather than in the f32 module (std::f32::EPSILON).
The new constants have existed since 1.43.0, which appears to be below the MSRV for the flatbuffers crate.
* Fixes incorrect key in flatbuffers Cargo.toml
The old code was using package.rust, which triggered a warning about an unused key:
warning: flatbuffers/rust/flatbuffers/Cargo.toml: unused manifest key: package.rust
The correct key for specifying MSRV is rust-version. See: https://doc.rust-lang.org/cargo/reference/rust-version.html#rust-version.
---------
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
* [Java] Use Table's fully qualified path
When a table's name is called `Table`, the Java bindings generated result in an error due to there being 2 Tables. This fixes the issue by fully qualifyng the flatbuffers Table import.
* Update codegen
* Update generated Java code
---------
Co-authored-by: Neville Dipale <neville@urbanlogiq.com>
* [Go] Write required string fields into the buffer when using Object API
In C++, CreateX allows to write the default "" value of a required
string, when the string is not explicitly set. Object API Pack method
uses this implementation of CreateX.
However, in go, despite whether the field is optional or required, it is
always checked against empty string allowing to create messages that
fail to pass the verifier. This commits partially reverts #7719 when the
string field is required.
* Add test for serializing required string fields using Object API
* Update generated code
The Monster schema contains a key string field. For historical
convenience reasons, string keys are assumed required.
* fix CScript string.compare
Because the default compare sorting rules of c# string are different from those of cpp, the binary file exported by flatc -b cannot use LookupByKey
* run generate_code.py
---------
Android libraries include <android/api-level.h> which defines the __ANDROID_API__ symbol even when targeting non-Android platforms and not using Android's libc. This updates the FLATBUFFERS_LOCALE_INDEPENDENT ifdef to check for __ANDROID__ before checking the Android API level.
Removes an extra check from the __Fuchsia__ branch. Fuchsia's libc does not support locales or the locale independent entry points.
Updates the Android API check to check for an API level >= 26 instead of 21. This matches the Android header file's availability macros and Bionic documentation:
https://android.googlesource.com/platform/bionic/+/HEAD/docs/status.md
This tried to generate from a directories "MyGame/Sample/"
for a empty path_ in M, MyGame & MyGame/Sample.
Which is incorrect since we want to start with the first
kPathSeparator `/` and not position 1.
We can't use macos-latest-large in non paid GitHub account but we can
use macos-15-intel in public repositories. If we use macos-15-intel,
we can run CI jobs for Intel macOS in fork repositories.
* Fixes to make SizeVerifier work.
In particular change all the places in the Flatbuffers library
and generated code that were using `Verifier` to instead use
`VerifierTemplate<TrackBufferSize>` and wrap them all inside
`template <bool TrackBufferSize = false>`.
Also add unit tests for SizeVerifier.
* Format using `sh scripts/clang-format-git.sh`
* Use `B` rather than `TrackBufferSize` for the name of the template parameter.
* Update generated files.
Implements InlineArrays which allow us to use Flatbuffers arrays within
Structs natively, and also implements FlatbufferVectors as a secondary API
when using mutable Structs
Fixes mutations within fixed sizes arrays
Adds tests and remove inout and mutating from generated objects in favor of borrowing
---------
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
* Support native_type for tables when using the C++ object API.
If native_type is specified on a table:
- No object API struct type is generated.
- The object API refers to the table by its native_type.
- UnPack and Create<TableName> methods are declared but not defined; as they
must be user-provided.
* Add tests for native_type on tables.
* Add documentation for native_type on tables.
Per the definition of --gen-compare: only generate comparators for object API
generated structs. Types annoated with native_type must define their own
comparators if `--gen-compare` is enabled.
Also enables --gen-compare for native_type_test and fixes the test by adding a
comparator for the Native::Vector3D type.
* Implements FlatbuffersVector in swift
Implements FlatbuffersVector which confirms to RandomAccessCollection,
this would give us semi-native sugary syntax to all the arrays in swift port.
This work will also be the foundation of using arrays in swift
* Fix failing tests for Swift
Previously: X::Pack forwarded to CreateX.
Now: CreateX will forward to X::Pack.
This is a step toward enabling using native types for tables when using the
object API. When defining a native table, the user will be able to define a
custom X::Pack method (which is more consistent with the existing native_type
functionality for structs). By reversing the order of the dependencies, CreateX
can continue to be auto-generated and will use the custom X::Pack method when
overriden for native_type tables.
* Prevent `make_span` from working with vectors and arrays of pointers
* support `make_structs_span` for little-endian
* fix build: add the required parentheses
---------
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
Treat flatbuffers' definition of npm_typescript as a dev dependency, in order to avoid conflicts when consuming flatbuffers in a repo that also depends on aspect_rules_ts.
* Correction of bug inside ToArray<T> methods
Avoid allocating too large buffers (len is expressed in bytes, not in Ts).
Added validation to ensure len is a multiple of SizeOf<T>() before converting to array.
* Update ByteBuffer.cs
* Refactor ToArray and ToArrayPadded methods
I understand from failed test that pos, len, padLeft and padRight are expressed in Ts
* Refactor ToArray and ToArrayPadded methods
* Final correction
All functions parameters expressed in bytes for homogeneity
Tests run:
- UNSAFE_BYTEBUFFER=true/ENABLE_SPAN_T=true: passed
- UNSAFE_BYTEBUFFER=true/ENABLE_SPAN_T=false: passed
- UNSAFE_BYTEBUFFER=false/ENABLE_SPAN_T=false: passed
- UNSAFE_BYTEBUFFER=false/ENABLE_SPAN_T=true: configuration forbidden by compilation
Correction of FlatBuffers.Test.csproj to allow UNSAFE_BYTEBUFFER/ENABLE_SPAN_T tests
Correction of FlatBuffersExampleTests.cs: I think the test was not run because it could not pass (to be reviewed carefully)
* Added ToSizedArrayPadded(int padLeft, int padRight) + ToArrayPadded(pos, len, padLeft, padRight) to the byteBuffers.
This is for API completion and to avoid unnecessary copy when framing my packets. I needed this to create a flat buffer with space in front of it for header / metadata.
* Fix indentation
---------
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
There are a couple instances where the ByteBuffer's Span property was accessed in a loop.
+ Extracted the use of the property outside of the loop to save a few cpu cycles.
Access to the allocator's internal buffer isn't exposed as a ReadOnlySpan<byte> from the ByteBuffer
or the FlatBufferBuilder.
+ Added a few convenience functions to access the buffer using a ReadOnlySpan<byte>.
There are a few cases where built in Span extensions can be used to run optimized code.
+ Added the use of Span.Fill() and ReadOnlySpan.SequenceCompareTo to replace existing loops.
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
new_p is a local addr but is owned now by slice_
thus the life time does not end at the end of the function
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
* fix(go/grpc): avoid panic on short FlatBuffers input
The gRPC codec read the root UOffsetT without checking input size. On
buffers shorter than SizeUOffsetT, GetUint32 touched data[3] and the
process panics.
Add a simple length check and validate the root offset stays within the
buffer. Return clear errors (insufficient data / invalid root offset)
instead of panicking.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
* fix(go/grpc): avoid signed overflow in offset
Keep the bounds check in the unsigned domain (UOffsetT) to avoid
signedness pitfalls.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
* chore: clarify comment regarding offset
A full FlatBuffer structure would be:
- uoffset_t: root table offset (4 bytes)
- soffset_t: vtable offset in root table (4 bytes)
- uint16_t: vtable size (2 bytes)
- uint16_t: table size (2 bytes)
In total 12 bytes. We are only validating the data length
before trying to read the uoffset_t, not the full structure.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
---------
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
Co-authored-by: Derek Bailey <derekbailey@google.com>
Our CI is broken and this is the error:
```
FAILURE: Build failed with an exception.
* What went wrong:
Gradle requires JVM 17 or later to run. Your build is currently configured to use JVM 11.
```
So updating our java-versions to the latest stable version which is 21 apparently.
Makes the return type of `static getFullyQualifiedName()` be a string literal instead of just the string type
Update tests
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
* [Python] Sync PythonTest.sh flags with generate_code.py
* [Python] Update generated code to latest flatc version for tests
* [Python] Fix test support for numpy newer than 2.0.0
* [Python] Remove unused variable
* [Python] Fix __eq__ for numpy arrays
* [Python] Run clang-format over the entire file
* Developers intro how to contribute
* Fix Rust code generation for Rust edition 2024
The errors look like:
```
warning[E0133]: call to unsafe function `fbs::flatbuffers::emplace_scalar` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::follow_cast_ref` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::Follow::follow` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::read_scalar_at` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::root_unchecked` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::size_prefixed_root_unchecked` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::Table::<'a>::new` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `std::slice::from_raw_parts` is unsafe and requires unsafe block
```
* Update goldens
Ran `goldens/generate_goldens.py`
* Regenerate code files
Ran `scripts/generate_code.py`
This commit significantly improves the developer experience for the Python Object-Based API by overhauling the generated `__init__` method for `T`-suffixed classes.
Previously, `T` objects had to be instantiated with an empty constructor, and their fields had to be populated manually one by one. This was verbose and not idiomatic Python.
This change modifies the Python code generator (`GenInitialize`) to produce `__init__` methods that are:
1. **Keyword-Argument-Friendly**: The constructor now accepts all table/struct fields as keyword arguments, allowing for concise, single-line object creation.
2. **Fully Typed**: The signature of the `__init__` method is now annotated with Python type hints. This provides immediate benefits for static analysis tools (like Mypy) and IDEs, enabling better autocompletion and type checking.
3. **Correctly Optional**: The generator now correctly wraps types in `Optional[...]` if their default value is `None`. This applies to strings, vectors, and other nullable fields, ensuring strict type safety.
The new approach remains **fully backward-compatible**, as all arguments have default values. Existing code that uses the empty constructor will continue to work without modification.
#### Example of a Generated `__init__`
**Before:**
```python
class KeyValueT(object):
def __init__(self):
self.key = None # type: str
self.value = None # type: str
```
**After:**
```python
class KeyValueT(object):
def __init__(self, key: Optional[str] = None, value: Optional[str] = None):
self.key = key
self.value = value
```
#### Example of User Code
**Before:**
```python
# Old, verbose way
kv = KeyValueT()
kv.key = "instrument"
kv.value = "EUR/USD"
```
**After:**
```python
# New, Pythonic way
kv = KeyValueT(key="instrument", value="EUR/USD")
```
Using the : syntax leads to non member attributes.
> If an attribute is defined in the class body with a type annotation
> but with no assigned value, a type checker should assume this is a non-member attribute
```
class Pet(Enum):
genus: str # Non-member attribute
species: str # Non-member attribute
CAT = 1 # Member attribute
DOG = 2 # Member attribute
```
https://typing.python.org/en/latest/spec/enums.html#defining-members
This prevents the include of the type defined in the pyi,
otherwise this leads to error message like this:
error: Name XYZ already defined (possibly by an import) [no-redef]
* [Python] Use correct type for str with None
Otherwise mypy will correctly flag code like this
def __init__(self):
self.fooBar = None # type: Optional[str]
error: Incompatible types in assignment (expression has type "None", variable has type "str")
* [Python] Make list type optional as they can contain None
Fixes an issue where exports were using incorrect relative paths for
>=3 namespace levels. This is fixed by making the starting range of the
namespace components relative to the amount of components.
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
Adding support for windows requires the code generations
to add a compiler statement to completely ignore GRPC code
generation on windows
Cleanup the project to use the main Package.swift to run tests
instead of having it separate and includes the imports for GRPC
within it.
Adds windows swift ci
* clang-format
* [Python] Replace . with _ in grpc filename suffix
Having filenames with . like `file.fb.grcp`
is not great for Python. Since dots are used for namespaces.
Replacing all of them with _ eg suffix `foo.bar.baz` will become
`foo_bar_baz`.
Restoring the previous default `_fb` suffix.
* [Python] Use namespace in path
This fixes a regression introduced with:
fb9afbafc7
And generates the grpc file in the namespace folder again.
* Sync commandline docs with web docs
* Offical Swift port for FlexBuffers
This is the offical port for FlexBuffers within
swift, and it introcudes a Common Module where code
is shared between flatbuffers and flexbuffers.
Writing most supported values like maps, vectors,
nil and scalars into a flexbuffer buffer. And includes
tests to verify that its similar to cpp
* Reading a flexbuffer
Implementing reading from a flexbuffer, enabling
most of the buffers features, like most types, maps, vectors,
typedvectors, and fixedtypedvectors.
Currently, if an offset/object cant be read we default to a swift
nil instead of the default flexbuffers 'null' with all values.
* Fixes bazel breaking due to new project structure
Address warnings within the library
* Adds comment on why we added the code & properly enforce the amout of bytes needed
For Vector or Array of structures the dereference operator of an
iterator returns the pointer to the structure. However, IndirectHelper,
which is used in the implementation of this operator, is instantiated
in the way that the IndirectHelper::Read returns structure by value.
This is because, Vector and Array instantiate IndirectHelper with
const T*, but VectorIterator instantiates IndirectHelper with T. There
are three IndirectHelper template definition: first for T, second for
Offset<T> and the last one for const T*. Those have different
IndirectHelper:Read implementations and (more importantly) return type.
This is the reason of mismatch in VectorIterator::operator* between
return type declaration and what was exactly returned.
That is, for Array<T,...> where T is scalar the VectorIterator is
instantiated as VectorIterator<T, T>, dereference operator returns T
and its implementation uses IndirectHelper<T> which Read function
returns T.
When T is not scalar, then VectorIterator is instantiated as
VectorIterator<T, const T *>, dereference operator returns const T * and
its implementation uses IndirectHelper<T> which Read function returns T.
The fix is done as follows:
* implement type trait is_specialization_of_Offset and
is_specialization_of_Offset64,
* change partial specialization of IndirectHelper with const T * that
it is instantiated by T and enabled only if T is not scalar and not
specialization of Offset or Offset64,
* remove type differentiation (due to scalar) from Array..
The above makes the IndirectHelper able to correctly instantiate itself
basing only on T. Thus, the instantiation in VectorIterator correctly
instantiate IndirectHelper::Read function, especially the return type.
Fixes access to union members when generating code with options "--cpp-field-case-style upper" and "--gen-object-api"
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
* Adds new API to reduce memory copying within swift
Adds new storage container _InternalByteBuffer which
will be holding the data that will be created within the swift
lib, however reading data will be redirected to ByteBuffer, which
should be able to handle all types of data that swift provide without
the need to copy the data itself. This is due to holding a reference to
the data.
Replaces assumingMemoryBinding with bindMemory which is safer
Adds function that provides access to a UnsafeBufferPointer for
scalars and NativeStructs within swift
Updates docs
Suppress compilation warnings by replacing var with let
Using overflow operators within swift to improve performance
Adds tests for GRPC message creation from a retained _InternalByteBuffer
* flatbuffers Rust reflection: replace num with num-traits
num crate is a wrapper over num-traits and a few other crates, that
reexports the APIs from all of them. We only need num-traits.
Signed-off-by: Marcin Radomski <dextero@google.com>
* Rust reflection: drop dependency on stdint crate
We only use it to get intmax_t for deriving alignment, which is an alias
for `core::ffi::c_long` [1]. We can use that directly instead.
[1] https://docs.rs/stdint/1.0.0/stdint/type.intmax_t.html
Signed-off-by: Marcin Radomski <dextero@google.com>
* Rust reflection: drop dependency on escape_string crate
It's used to format a string used for debugging only, so we might as
well use the builtin Debug representation of a string.
Signed-off-by: Marcin Radomski <dextero@google.com>
* Rust codegen: add derives on generated bitflags
Otherwise it limits the use of structs generated for reflection.fbs
in Rust reflection API.
Signed-off-by: Marcin Radomski <dextero@google.com>
* Rust flatbuffers: update bitflags dependency to 2.8
Signed-off-by: Marcin Radomski <dextero@google.com>
* Rust codegen: use bitflags v2 API for converting from bits
from_bits_unchecked was replaced with safe from_bits_retain.
Signed-off-by: Marcin Radomski <dextero@google.com>
* Regenerate Rust code after idl change
Signed-off-by: Marcin Radomski <dextero@google.com>
* Regenerate reflection_generated.rs
With flatc --rust ../../../reflection/reflection.fbs
Signed-off-by: Marcin Radomski <dextero@google.com>
* ts/BUILD.bazel: add missing import
Found by Buildifire presubmit:
Function "sh_binary" is not global anymore and needs to be loaded from
"@rules_shell//shell:sh_binary.bzl".
Signed-off-by: Marcin Radomski <dextero@google.com>
* Update expected value in generated_code_debug_prints_correctly test
In bitflags v2, the debug string representation of enum values is
different than it was in v1:
Blue -> Color(Blue)
(empty) -> LongEnum(0x0)
This change adjusts the expected test value.
Signed-off-by: Marcin Radomski <dextero@google.com>
* Fix tests build on Swift 5.8
grpc-swift 1.4.1 depends on swift-nio-ssl 2.14.0+ [1]. swift-nio-ssl 2.29.1
published on 2025-01-30, introduced some code [2] that uses a "switch
expression syntax" supported since Swift 5.9 [3]. Attempts to compile it with
Swift 5.8 cause build errors.
swift-nio-ssl project doesn't seem to support Swift 5.8. A commit from
2024-10-29 removes a "deprecated reference to a Swift 5.8 pipeline" [4].
swift-nio-ssl 2.29.0 is the last version that can be compiled with Swift
5.8. This commit pins it to that exact version.
[1] 66e27d7e84/Package.swift (L33)
[2] 3cb4d5ad12 (diff-bc1db1321ff689c2819245dcce1a3080554f0fc13f81b8d326c97e7d42717c8fR54)
[3] https://github.com/swiftlang/swift-evolution/blob/main/proposals/0380-if-switch-expressions.md
[4] 8a6b89d9a4
---------
Signed-off-by: Marcin Radomski <dextero@google.com>
Co-authored-by: Marcin Radomski <dextero@google.com>
This setup is much simpler than calling Bazel from within Bazel
and making sure files and flags are set up correctly.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Add missing file to filegroup for bazel integration tests
Fixup after a9257b6963.
* Align versions in bazel_respository_test_dir with root
* Update XCode version to 15.2
This is the oldest available version.
* Add WORKSPACE.bzlmod
* Add support for Bazel 7
* Add support for Bazel 8 in CI
* Restrict visibility of exported file
* Align npm_translate_lock attrs
* Remove defs_bzl_filename attr
* Align root_package with pnpm-lock.yaml location
Use a symlink to avoid copying the file.
* Update Buildkite Bazel CI
Restructure presubmit.yml to support matrix.
* Remove testing Ubuntu 18.04
The available LLVM 6.0 is too old to support std::filesystem.
* Add testing in Ubuntu 22.04
* Use Bazel version 6.5.0 in integration test
We need to copy the .fbs files into the package used for .bfbs files.
This is necessary as flatc doesn't provide support to specify the full
output file name for an .fbs file in a different folder.
I tried OUTPUT_FILE env var but this doesn't seem to be honored by
flatc.
Transitive headers like array.h have not been available in the runtime_cc target causing the build to fail. Adding all public headers to make sure transitive headers of flatbuffers.h are available.
* #Rust Create a crate for reflection
* #Rust Add a crate for reflection tests and helper to access schema
* #Rust Get root table of a buffer and access field with schema
* #Rust Add 'Struct' struct and corresponding getter
* #Rust Add functions of getting any table/struct field value as integer/float/string
* #Rust Add setters for scalar fields
* #Rust Add setter for string fields
* #Rust Add getter for Table/Vector fields
* #Rust Add buffer verification
* Add a 'SafeBuffer' struct which provides safe methods for reflection
It verifies buffer against schema during construction and provides all the unsafe getters in lib.rs in a safe way
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Improves vectors performance and adds a benchmark to vectors of offsets in swift
Improves performance for all arrays and for loops
Uses a tuple instead of allocating a struct each time we start iterating over fieldloc
Updates benchmark library
* Fixing swift Wasm ci
* Update grpc-core version
io.grpc:grpc-core package in version 1.36.0 contains multiple [CVE's](https://mvnrepository.com/artifact/io.grpc/grpc-core/1.36.0).
Bump grpc-core version to latest 1.68.0 version to mitigate potential vulnerabilities.
* Update grpc version to 1.67.1
grpc was mistakenly released to maven under version 1.68.0 whenever a real release was done for version 1.67.1 [1]. The mistake was fixed later.
[1] https://github.com/grpc/grpc-java/releases
Fix CVE-2022-25647
The package com.google.code.gson:gson before 2.8.9 is vulnerable to Deserialization of Untrusted Data via the writeReplace() method in internal classes, which may lead to denial of service attacks.
Bump up version of the gson package.
https://github.com/advisories/GHSA-4jrv-ppp4-jm57
This allows this type to meet the requirements of e.g.
std::ranges::range, which is necessary for it to work with the
std::span range constructor, or the "non-legacy" constructor for
Chromium's base::span.
Bug: none
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Update NativeObject.swift
Correct the word.
* Update ByteBuffer.swift
Type parameter does not existing, remove it.
* Update ByteBuffer.swift
Correct the word.
In case when flatbuffers are being used along with other project that defines "max" preprocessor macro, the ::max() in FLATBUFFERS_MAX_BUFFER_SIZE and FLATBUFFERS_MAX_64_BUFFER_SIZE is incorrectly being expanded to the macro. Adding parentheses enforces function-like interpretation.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Move `namer.h` and `idl_namer.h` to `include/codegen` so they can be reused from `grpc` dirqectory.
* [gRPC] Update the Python generator to produce typed handlers and Python stubs if requested.
* [gRPC] Document the newly added compiler flags.
This allows enums to be type check with mypy.
They will still behave like ints ->
> IntEnum is the same as Enum,
> but its members are also integers and can be used anywhere
> that an integer can be used.
> If any integer operation is performed with an IntEnum member,
> the resulting value loses its enumeration status.
https://docs.python.org/3/library/enum.html#enum.IntEnum
Only if the --python-typing flag is set.
* [Python] Generate `.pyi` stub files when `--python-typing` is on.
To support this change, the following modifications were made:
- added a new option to disable `numpy` helpers generation;
- added a new flag to control the target Python version:
`--python-version` can be one of the following:
- `0.x.x` – compatible with any Python version;
- `2.x.x` – compatible with Python 2;
- `3.x.x` – compatible with Python 3.
- added codegen utilities for Python;
- added a note that the generated .py file is empty.
* [C++] Update the validator to skip structs in namespaces other than the current one.
Updates copyright from 2023 to 2024 & formats code - updates formatting rules
Updates CI to run with swift 5.8
Adds wasmer & updates command to run carton as a swift plugin
Update bazelci to also accept swift 5.8
Adds swift 5.10 to the test matrix
* [Python] Generate `.pyi` stub files when `--python-typing` is on.
To support this change, the following modifications were made:
- added a new option to disable `numpy` helpers generation;
- added a new flag to control the target Python version:
`--python-version` can be one of the following:
- `0.x.x` – compatible with any Python version;
- `2.x.x` – compatible with Python 2;
- `3.x.x` – compatible with Python 3.
- added codegen utilities for Python;
- added a note that the generated .py file is empty.
* [Python] Update Bazel build rules.
* [Python] Update Bazel build rules.
* [Python] Run buildifier on BUILD.bazel files.
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
This will allow the code to be compiled with `-Wfloat-equal`
as this would result in the folowing warning/error:
vendor/flatbuffers/include/flatbuffers/base.h:465:69:
error: comparing floating point with == or != is unsafe [-Werror,-Wfloat-equal]
template<typename T> inline bool IsTheSameAs(T e, T def) { return e == def; }
But the way it is used in flatbuffers it is ok to compare floating
points with ==.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Fixes Bazel issues for windows and ci
Fetching boringssl within the flatbuffers repository, to patch the issues
of not being able to upgrade to Xcode 14.3 due to buildkite throwing
errors. The patch was inspired by the tenserflow patch
https://github.com/tensorflow/tensorflow/issues/60191#issuecomment-1496073147
Removes references of swift from the windows pipeline for bazel
Sets github actions to use xcode 14.3 for kotlin and sets the macOS
build for intel cpus.
* Update build.yml
Remove comment that is not relevant any longer.
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Update build.yml to ubuntu-24.04
Apparently g++-13 was removed from the ubuntu-22.04 runners.
We also don't have enterprise runners at 24.04 yet, so just use the free ones for now until we get support for those. CI builds might take longer now.
* Update build.yml
Downgrade to g++12 and revert change to using normal runners
* Update build.yml
Go back to ubuntu-24.04 and update both gcc and clang to their latest versions according to [this](https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2404-Readme.md?plain=1#L16-L20).
* Update build.yml
Go back to g++13 for now, as we get some exotic warning in g++14 for newer C++ standards.
* Update build.yml
Fix the other issues with `macos-latest` going to arm: https://github.com/actions/runner-images/tree/main?tab=readme-ov-file#available-images and that Swift wasn't installed in the ubuntu-24.04 by default.
* Update build.yml
Disable Kotlin MacOs CI
The new options are:
- `--grpc-filename-suffix` controls the suffix of the generated files;
- `--grpc-use-system-headers` controls the type of C++ includes generated;
- `--grpc-search-path` controls the directory that contains gRPC runtime;
- `--grpc-additional-header` allows to provide additional dependencies for the generated code.
* [Python] Fix various codegen problems.
This includes:
- escaping keywords happens **after** converting the case:
- currently, `table ClassT` generate `class = Class()` which is invalid Python;
- imports in `one_file` mode use the filename rather than the type name when resolving module names;
- use `filename_suffix` instead of the hardcoded `_generated` one;
- generate empty files if no structs or enums are available. This makes the set of output files more predictable for Bazel.
* [Python] Fix various codegen problems.
This includes:
- escaping keywords happens **after** converting the case:
- currently, `table ClassT` generate `class = Class()` which is invalid Python;
- imports in `one_file` mode use the filename rather than the type name when resolving module names;
- use `filename_suffix` instead of the hardcoded `_generated` one;
- generate empty files if no structs or enums are available. This makes the set of output files more predictable for Bazel.
When flatbuffers is being used from a project that has no use for
JavaScript, users encounter an error similar to the following:
ERROR: Skipping '@com_github_google_flatbuffers//:flatbuffers': error loading package '@com_github_google_flatbuffers//': Unable to find package for @npm//:defs.bzl: The repository '@npm' could not be resolved: Repository '@npm' is not defined.
WARNING: Target pattern parsing failed.
ERROR: error loading package '@com_github_google_flatbuffers//': Unable to find package for @npm//:defs.bzl: The repository '@npm' could not be resolved: Repository '@npm' is not defined.
INFO: Elapsed time: 0.023s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
currently loading: @com_github_google_flatbuffers//
That's not ideal. Users that only care about C++ for example
shouldn't be forced to deal with rules_js and friends.
This patch attempts to fix that by moving the rules_js-specific things
into the `ts` and `tests/ts` directories. This should allow
non-JavaScript projects to ignore rules_js and friends completely.
Here I basically followed the `rules_foo` example from rules_js:
https://github.com/aspect-build/rules_js/tree/main/e2e/rules_foo
The idea is that flatbuffers has its own npm dependencies regardless
of what other projects may have. This means we should not force the
user to import flatbuffers's npm dependencies. The new
`ts/repositories.bzl` file is used by dependents to import
flatbuffers's dependencies. They can still import their own
dependencies. This cleanup allowed me to move all
JavaScript-specific stuff from the top-level directory into
subdirectories.
There should be no changes in this patch in terms of functionality.
It's just a refactor of the rules_js call sites. Users will have to
add a call to the function in `ts/repositories.bzl` in their own
`WORKSPACE` file. They can use
`tests/ts/bazel_repository_test/WORKSPACE` as an example.
Co-authored-by: Derek Bailey <derekbailey@google.com>
If a schema contains a message named e.g. FooT and a message named Foo
while the Object API suffix is T, then two classes with colliding names
will be generated. This scenario will produce a C++ compiler error, but
it's confusing.
This patch moves the error to the compiler, allowing the user to more
readily act to correct the issue.
Co-authored-by: Michael Beardsworth <beardsworth@intrinsic.ai>
Fix "One Definition Rule" violation when using flatbuffers::Verifier with
FLATBUFFERS_TRACK_VERIFIER_BUFFER_SIZE defined in some compilation units
and not defined in other compilation units.
The fix is to make Verifier a template class, with a boolean template
parameter replacing the "#ifdef" conditionals; to rename it as
VerifierTemplate; and then to use "#ifdef" only for a "using" declaration
that defines the original name Verifier an an alias for the instantiated
template. In this way, even if FLATBUFFERS_TRACK_VERIFIER_BUFFER_SIZE is
defined in some compilation units and not in others, as long as clients
only reference flatbuffers::Verifier in .cc files, not header files, there
will be no ODR violation, since the only part whose definition varies is the
"using" declaration, which does not have external linkage.
There is still some possibility of clients creating ODR violations
if the client header files (rather than .cc files) reference
flatbuffers::Verifier. To avoid that, this change also deprecates
FLATBUFFERS_TRACK_VERIFIER_BUFFER_SIZE, and instead introduces
flatbuffers::SizeVerifier as a public name for the template instance with
the boolean parameter set to true, so that clients don't need to define
the macro at all.
* Reproduce the error in a unit test
Reproduces #8200
* Overload KeyCompareWithValue to work for string-like objects
This fixes#8200.
* Extra tests
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
It appears the upgrade to xcode 14.3 broke the macos build on builkite.
The last good build was using xcode 14.2, so go back to this version
until the issue is resolved.
* flatc: `--grpc-callback-api` flag generates C++ gRPC Callback API server `CallbackService` skeletons AND client native callback/async stubs (unary + all streaming reactor forms) (opt-in, non-breaking, issue #8596).
* Swift - Adds new API to reduce memory copying within swift (#8484)
* Rust - Support Rust edition 2024 (#8638)
* [C++] - Use the Google Style for clang-format without exceptions (#8706)
