* Migrating from Xctests to swift testing
This migrates to the new Swift testing framework,
which would allow us to always use the latest tech
from swift moving forward.
* Updates flag to make sure that Wasm testing works
* Migrate to swift 6.0 & swift-gRPC 2.0
The following migrates to swift 6.0, and also
migrate to swift-grpc 2.0 that uses swift-nio
under the hood to provide nicer API and async await
Adds sendable to enum & update @_implementationOnly imports to use internal imports
* Address PR comments regarding misspelling & proper method naming.
VerifyKey() returns true on the first non-zero byte instead of
checking for a null terminator. This causes VerifyBuffer() to accept
FlexBuffers with non-null-terminated keys. Subsequent access to those
keys via strlen()/strcmp() reads out of bounds.
The condition if (*p++) should be if (!*p++) — return true
when a null terminator is found, not when any non-zero byte is found.
Confirmed with AddressSanitizer: heap-buffer-overflow in strlen()
after VerifyBuffer() returns true on a corrupted buffer.
Copy the repo-root LICENSE into the Python package directory for the duration of setup() so license_files = LICENSE remains valid without using deprecated parent-directory paths.
Remove the staged copy after the build completes.
* fix: correct operator precedence in ForAllFields reverse iteration
The expression `size() - i + 1` evaluates as `(size() - i) + 1` due to
left-to-right associativity, producing an out-of-bounds index when
reverse=true. For a vector of size N, the first iteration (i=0) accesses
index N+1, which is 2 past the last valid index.
Changed to `size() - (i + 1)` to match the correct implementation
already present in bfbs_gen.h:192.
Bug: CWE-125 (Out-of-bounds Read), CWE-783 (Operator Precedence Error)
* test: add ForAllFieldsReverseTest for reverse iteration correctness
Verify that ForAllFields with reverse=true iterates fields in
descending ID order. Tests both Stat (3 fields) and Monster
(many fields with non-sequential definition order) tables.
---------
Co-authored-by: Tulgaa <tulgaa.kek@gmail.com>
* Add fallible try_* API for FlatBufferBuilder
This is to support error propagation from Allocator trait. The Allocator
grow_downwards() method returns Result<(), Self::Error>, but
FlatBufferBuilder panics via .expect() when allocation fails instead of
propagating the error.
* Add rust fallible API docs
A blob is an array of bytes and has no intrinsic alignment (i.e. the
alignment is 1). The alignment of the existing flexbuffers blob is
solely affected by the width of the integer needed to store the blob's
size: that integer's width becomes the alignment of the blob.
The proposed AlignedBlob function here piggybacks on this effect and
simply uses a user-defined alignment for the width of the integer that
stores the blob's size; this automatically imparts that same alignment
on the blob itself. (The width is bounded below by the actual width
needed to store the blob's size.)
The ability to control the alignment of a blob is important for use
cases in which the blob itself stores structured data that we want to
access without further copies (e.g. other flatbuffer messages).
String default values parsed from .fbs schemas are un-escaped by the IDL
parser (e.g., \x22 becomes a raw " byte), but code generators embed these
raw values directly into generated source code string literals. This allows
specially crafted .fbs files to break out of string literals and inject
arbitrary code into generated C++, Rust, TypeScript, and Swift source.
Fix by adding EscapeCodeGenString() helper that re-escapes string content
before embedding, and applying it to all 7 affected injection points across
5 code generators (C++, Rust, TypeScript, Swift, FBS).
Resolves the TODO comments in idl_gen_cpp.cpp and idl_gen_rust.cpp.
* fix(flatbuffers): use manual impl Default for struct object types
* fix: handle bool and float zero literals in struct object Default impl
* fix: regenerate all test bindings with generate_code.py
* fix: data type check on swift build
* fix: test large array on struct and enum
When building a FlexBuffer using the Builder and adding data that exceeds the default buffer size (2048 bytes), in _newOffset() a larger buffer is created, but never used. This results in a RangeError.
Resolve by actually replacing the too small with the new larger buffer. Add a test that verifies this by adding multiple large strings to a vector.
There is a typo in the schema.md file in documentation. An empty 'required' section was added right before the 'required' item in the middle of the attributes list. It even appears in the table of content, which might confuse readers, making it look like following attributes might be required while they are not. You can notice the issue there: https://flatbuffers.dev/schema/#attributes.
I did check that `mkdocs serve -f mkdocs.yml` does produce the expected output (the same attributes list without that extra empty `required` section in the middle) with my changes.
* [C#] Add GetBytes methods for fixed arrays
I wanted to direct access to fixed array bytes. I made some changes to the idl generator to create GetBytes functions following the same naming conventions used for vectors of scalar types. There was not a 'Length' field present to bound the existing index accessor so I added that too.
+ Add generic GetBytes for fixed length arrays of scalar types
+ Implement conditional compilation for ENABLE_SPAN_T:
- ENABLE_SPAN_T: Returns `Span<T>` using `MemoryMarshal.Cast<byte, T>()` as needed.
- Else: Returns `ArraySegment<byte>?` for raw byte access
+ Added tests reusing arrays_test.fbs definitions
+ Added const int Length field to support existing index based accessors
* [C#] Sync generated code for after adding GetBytes methods for fixed arrays
---------
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
* Adds tests for Rust single file mode
All existing tests only compile Rust code using --rust-module-root-file.
* Adds standalone include tests for Rust
The imports for these tests have been moved to their own file, as the existing intergration_test.rs file hides compilation issues from code generation due to symbols brought into scope outside of the generated code (e.g. `extern crate alloc`).
* Declare alloc crate in every Rust namespace
When performing code generation within a single file, extern crate alloc needs to be delcared to bring alloc into scope within every inner namespace.
* Regenerates generated schemas
The write_vtable() function's comment claimed to "fill the WIP vtable
with zeros" but make_space() only reserves memory without initializing
it. When using custom allocators with non-zeroed buffers, unset vtable
field entries would contain garbage instead of zero (which indicates
"use default value").
This fix explicitly zeros the vtable memory after reserving space,
matching the C++ implementation's buf_.fill_big() behavior.
Added regression test using a garbage-filled allocator (0xAA) that
verifies vtable entries for unset fields are properly zeroed.
Fixes#8894
* Add --ts-undefined-for-optionals command line option
# Details
- Fixes#7656
- Added a new `--ts-undefined-for-optionals` command line option for `flatc`.
- If enabled, generated TypeScript code uses `undefined` for optional fields rather than `null`.
* Also add TS generated test files
* Run `sh scripts/clang-format-git.sh`
* also add tests/ts/lalala-options.ts to the repo
* move new tests to tests/ts/optional_values dir
* add tests/ts/optional_values/optional_values_generated.cjs to the repo
* reuse existing optional_scalars.fbs and add new test
* add comma
* sh scripts/clang-format-git.sh
* remove comma
* sh scripts/clang-format-git.sh
* trying things
* sh scripts/clang-format-git.sh
* done
* address feedback
* sh scripts/clang-format-git.sh
* run `sh scripts/clang-format-git.sh`
* remove uneeded `eslint-disable @typescript-eslint/no-namespace` line
---------
Co-authored-by: José Luis Millán <jmillan@aliax.net>
* fix for https://github.com/google/flatbuffers/issues/8759
__vector_as_array<T> calling ByteBuffer.ToArray<T> with the length in bytes by multiplying len with ByteBuffer.Sizeof<T> and FlatBuffersExampleTests extended to call GetVectorOfLongsArray/GetVectorOfDoublesArray which failed without the fix
* first try to repair build-dotnet-windows
* syntax error fixed
* Update solution creation command in build workflow
add --format sln to the dotnet new command, maybe it is currently creating a .slnx instead?
* Fix generate_code script path
* [Python] Make StartVector public
Make StartVector vector public since it is already being used in
generated code
* [Python] Improve vector creation for Python API
Makes Python API for vectors cleaner like Rust and Swift
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
* chore(idl): Check for case insensitive keywords
Most languages are not affected by this change. In PHP, some names such
as Bool cannot be used because it is a reserved keyword for to the bool
data type. The field `keywords_casing` in the configs enables checking
all characters in lowercase against every keyword. This should be safe
as flatbuffers does not allow non-ASCII characters in its grammar.
* chore: Fix formatting to follow google's coding style for enums
* chore: Extract convert case to lower when CaseInsensitive
---------
Co-authored-by: Justin Davis <jtdavis777@gmail.com>
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Remove invalid dependency on FLATBUFFERS_GENERATE_HEADERS_SCHEMAS
add_dependencies() is for targets.
CMake 4.2.0 fails because of this (it shouldn't crash though, but that's another topic). See https://gitlab.kitware.com/cmake/cmake/-/issues/27415
* Use FLATC_TARGET
---------
Co-authored-by: Justin Davis <jtdavis777@gmail.com>
* generate mutable union accessors
* add test
* Revert "add test"
This reverts commit 45e352b18f.
* update file
* formatter got in the way
* merge conflicts
* updated genned code
* manually fix code gen bc I can't figure out why this file won't code gen
---------
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
Add missing bounds checking to ByteVector before slice
operations in the Go FlatBuffers implementation. Relative offset and
vector length are now checked against the buffer size. Instead of
panicking, the code now returns nil. Regression test added.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
Co-authored-by: Justin Davis <jtdavis777@gmail.com>
* arrays of enums with no value for 0 now throw errors
* move setting key field outside struct check
* set to default instead of required
* unsure of why these bfbs files have changed at this time, checking them in to run the pipelines.
* remove known bad test
* flatc builds and seems to work, some of the extra targets are having linker errors
* fix build system
* pipeline failures
* un-rename files
* refactor to use unique_ptr
* typo
* rm make_unique, add comments
* fix cmake
---------
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
Improves unpack performance for vectors by allowing the compiler
to vectorize flatbuffers::Vector to std::vec::Vec conversions,
using the unstable trusted_len feature.
Internally, enables an optimization in src/alloc/vec/spec_extend.rs:
Previously, unpacking a flatbuffers::Vector called
SpecExtend::extend_desugared fallback, which inhibits vectorization
(due to a branch before every element move). Declaring TrustedLen
allows SpecExtend::extend_trusted, which LLVM can often vectorize
into a memcpy.
For [ubyte] vectors in particular, this turns a rather expensive
loop of 'mov BYTE PTR [rax+r13*1], bpl' into a `call memcpy`.
* add deser support for enum type
* update generated files
* remove deser generator when bitflag enable
* add deser test
* Restore the Rust editions version
* Remove unnecessary modifications
* Fixes checks for serde features in flexbuffers crate
* Removes unused MapReaderIndexer use statement
* Fixes warning about nightly cfg usage
Enabling a cfg attribute through cargo::rustc-cfg in build.rs should be coupled with a cargo::rust-check-cfg value so that the compiler knows about the custom cfg. See: https://doc.rust-lang.org/rustc/check-cfg/cargo-specifics.html#cargorustc-check-cfg-for-buildrsbuild-script.
* Migrates usage of deprecated float constants
This update fixes a compiler warning from use of the old constants.
Constants like EPSILON are now directly on the float primitives (e.g. f32::EPSILON) rather than in the f32 module (std::f32::EPSILON).
The new constants have existed since 1.43.0, which appears to be below the MSRV for the flatbuffers crate.
* Fixes incorrect key in flatbuffers Cargo.toml
The old code was using package.rust, which triggered a warning about an unused key:
warning: flatbuffers/rust/flatbuffers/Cargo.toml: unused manifest key: package.rust
The correct key for specifying MSRV is rust-version. See: https://doc.rust-lang.org/cargo/reference/rust-version.html#rust-version.
---------
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
* [Java] Use Table's fully qualified path
When a table's name is called `Table`, the Java bindings generated result in an error due to there being 2 Tables. This fixes the issue by fully qualifyng the flatbuffers Table import.
* Update codegen
* Update generated Java code
---------
Co-authored-by: Neville Dipale <neville@urbanlogiq.com>
* [Go] Write required string fields into the buffer when using Object API
In C++, CreateX allows to write the default "" value of a required
string, when the string is not explicitly set. Object API Pack method
uses this implementation of CreateX.
However, in go, despite whether the field is optional or required, it is
always checked against empty string allowing to create messages that
fail to pass the verifier. This commits partially reverts #7719 when the
string field is required.
* Add test for serializing required string fields using Object API
* Update generated code
The Monster schema contains a key string field. For historical
convenience reasons, string keys are assumed required.
* fix CScript string.compare
Because the default compare sorting rules of c# string are different from those of cpp, the binary file exported by flatc -b cannot use LookupByKey
* run generate_code.py
---------
Android libraries include <android/api-level.h> which defines the __ANDROID_API__ symbol even when targeting non-Android platforms and not using Android's libc. This updates the FLATBUFFERS_LOCALE_INDEPENDENT ifdef to check for __ANDROID__ before checking the Android API level.
Removes an extra check from the __Fuchsia__ branch. Fuchsia's libc does not support locales or the locale independent entry points.
Updates the Android API check to check for an API level >= 26 instead of 21. This matches the Android header file's availability macros and Bionic documentation:
https://android.googlesource.com/platform/bionic/+/HEAD/docs/status.md
This tried to generate from a directories "MyGame/Sample/"
for a empty path_ in M, MyGame & MyGame/Sample.
Which is incorrect since we want to start with the first
kPathSeparator `/` and not position 1.
We can't use macos-latest-large in non paid GitHub account but we can
use macos-15-intel in public repositories. If we use macos-15-intel,
we can run CI jobs for Intel macOS in fork repositories.
* Fixes to make SizeVerifier work.
In particular change all the places in the Flatbuffers library
and generated code that were using `Verifier` to instead use
`VerifierTemplate<TrackBufferSize>` and wrap them all inside
`template <bool TrackBufferSize = false>`.
Also add unit tests for SizeVerifier.
* Format using `sh scripts/clang-format-git.sh`
* Use `B` rather than `TrackBufferSize` for the name of the template parameter.
* Update generated files.
Implements InlineArrays which allow us to use Flatbuffers arrays within
Structs natively, and also implements FlatbufferVectors as a secondary API
when using mutable Structs
Fixes mutations within fixed sizes arrays
Adds tests and remove inout and mutating from generated objects in favor of borrowing
---------
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
* Support native_type for tables when using the C++ object API.
If native_type is specified on a table:
- No object API struct type is generated.
- The object API refers to the table by its native_type.
- UnPack and Create<TableName> methods are declared but not defined; as they
must be user-provided.
* Add tests for native_type on tables.
* Add documentation for native_type on tables.
Per the definition of --gen-compare: only generate comparators for object API
generated structs. Types annoated with native_type must define their own
comparators if `--gen-compare` is enabled.
Also enables --gen-compare for native_type_test and fixes the test by adding a
comparator for the Native::Vector3D type.
* Implements FlatbuffersVector in swift
Implements FlatbuffersVector which confirms to RandomAccessCollection,
this would give us semi-native sugary syntax to all the arrays in swift port.
This work will also be the foundation of using arrays in swift
* Fix failing tests for Swift
Previously: X::Pack forwarded to CreateX.
Now: CreateX will forward to X::Pack.
This is a step toward enabling using native types for tables when using the
object API. When defining a native table, the user will be able to define a
custom X::Pack method (which is more consistent with the existing native_type
functionality for structs). By reversing the order of the dependencies, CreateX
can continue to be auto-generated and will use the custom X::Pack method when
overriden for native_type tables.
* Prevent `make_span` from working with vectors and arrays of pointers
* support `make_structs_span` for little-endian
* fix build: add the required parentheses
---------
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
Treat flatbuffers' definition of npm_typescript as a dev dependency, in order to avoid conflicts when consuming flatbuffers in a repo that also depends on aspect_rules_ts.
* Correction of bug inside ToArray<T> methods
Avoid allocating too large buffers (len is expressed in bytes, not in Ts).
Added validation to ensure len is a multiple of SizeOf<T>() before converting to array.
* Update ByteBuffer.cs
* Refactor ToArray and ToArrayPadded methods
I understand from failed test that pos, len, padLeft and padRight are expressed in Ts
* Refactor ToArray and ToArrayPadded methods
* Final correction
All functions parameters expressed in bytes for homogeneity
Tests run:
- UNSAFE_BYTEBUFFER=true/ENABLE_SPAN_T=true: passed
- UNSAFE_BYTEBUFFER=true/ENABLE_SPAN_T=false: passed
- UNSAFE_BYTEBUFFER=false/ENABLE_SPAN_T=false: passed
- UNSAFE_BYTEBUFFER=false/ENABLE_SPAN_T=true: configuration forbidden by compilation
Correction of FlatBuffers.Test.csproj to allow UNSAFE_BYTEBUFFER/ENABLE_SPAN_T tests
Correction of FlatBuffersExampleTests.cs: I think the test was not run because it could not pass (to be reviewed carefully)
* Added ToSizedArrayPadded(int padLeft, int padRight) + ToArrayPadded(pos, len, padLeft, padRight) to the byteBuffers.
This is for API completion and to avoid unnecessary copy when framing my packets. I needed this to create a flat buffer with space in front of it for header / metadata.
* Fix indentation
---------
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
There are a couple instances where the ByteBuffer's Span property was accessed in a loop.
+ Extracted the use of the property outside of the loop to save a few cpu cycles.
Access to the allocator's internal buffer isn't exposed as a ReadOnlySpan<byte> from the ByteBuffer
or the FlatBufferBuilder.
+ Added a few convenience functions to access the buffer using a ReadOnlySpan<byte>.
There are a few cases where built in Span extensions can be used to run optimized code.
+ Added the use of Span.Fill() and ReadOnlySpan.SequenceCompareTo to replace existing loops.
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
new_p is a local addr but is owned now by slice_
thus the life time does not end at the end of the function
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
* fix(go/grpc): avoid panic on short FlatBuffers input
The gRPC codec read the root UOffsetT without checking input size. On
buffers shorter than SizeUOffsetT, GetUint32 touched data[3] and the
process panics.
Add a simple length check and validate the root offset stays within the
buffer. Return clear errors (insufficient data / invalid root offset)
instead of panicking.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
* fix(go/grpc): avoid signed overflow in offset
Keep the bounds check in the unsigned domain (UOffsetT) to avoid
signedness pitfalls.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
* chore: clarify comment regarding offset
A full FlatBuffer structure would be:
- uoffset_t: root table offset (4 bytes)
- soffset_t: vtable offset in root table (4 bytes)
- uint16_t: vtable size (2 bytes)
- uint16_t: table size (2 bytes)
In total 12 bytes. We are only validating the data length
before trying to read the uoffset_t, not the full structure.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
---------
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
Co-authored-by: Derek Bailey <derekbailey@google.com>
Our CI is broken and this is the error:
```
FAILURE: Build failed with an exception.
* What went wrong:
Gradle requires JVM 17 or later to run. Your build is currently configured to use JVM 11.
```
So updating our java-versions to the latest stable version which is 21 apparently.
Makes the return type of `static getFullyQualifiedName()` be a string literal instead of just the string type
Update tests
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
* [Python] Sync PythonTest.sh flags with generate_code.py
* [Python] Update generated code to latest flatc version for tests
* [Python] Fix test support for numpy newer than 2.0.0
* [Python] Remove unused variable
* [Python] Fix __eq__ for numpy arrays
* [Python] Run clang-format over the entire file
* Developers intro how to contribute
* Fix Rust code generation for Rust edition 2024
The errors look like:
```
warning[E0133]: call to unsafe function `fbs::flatbuffers::emplace_scalar` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::follow_cast_ref` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::Follow::follow` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::read_scalar_at` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::root_unchecked` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::size_prefixed_root_unchecked` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::Table::<'a>::new` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `std::slice::from_raw_parts` is unsafe and requires unsafe block
```
* Update goldens
Ran `goldens/generate_goldens.py`
* Regenerate code files
Ran `scripts/generate_code.py`
This commit significantly improves the developer experience for the Python Object-Based API by overhauling the generated `__init__` method for `T`-suffixed classes.
Previously, `T` objects had to be instantiated with an empty constructor, and their fields had to be populated manually one by one. This was verbose and not idiomatic Python.
This change modifies the Python code generator (`GenInitialize`) to produce `__init__` methods that are:
1. **Keyword-Argument-Friendly**: The constructor now accepts all table/struct fields as keyword arguments, allowing for concise, single-line object creation.
2. **Fully Typed**: The signature of the `__init__` method is now annotated with Python type hints. This provides immediate benefits for static analysis tools (like Mypy) and IDEs, enabling better autocompletion and type checking.
3. **Correctly Optional**: The generator now correctly wraps types in `Optional[...]` if their default value is `None`. This applies to strings, vectors, and other nullable fields, ensuring strict type safety.
The new approach remains **fully backward-compatible**, as all arguments have default values. Existing code that uses the empty constructor will continue to work without modification.
#### Example of a Generated `__init__`
**Before:**
```python
class KeyValueT(object):
def __init__(self):
self.key = None # type: str
self.value = None # type: str
```
**After:**
```python
class KeyValueT(object):
def __init__(self, key: Optional[str] = None, value: Optional[str] = None):
self.key = key
self.value = value
```
#### Example of User Code
**Before:**
```python
# Old, verbose way
kv = KeyValueT()
kv.key = "instrument"
kv.value = "EUR/USD"
```
**After:**
```python
# New, Pythonic way
kv = KeyValueT(key="instrument", value="EUR/USD")
```
Using the : syntax leads to non member attributes.
> If an attribute is defined in the class body with a type annotation
> but with no assigned value, a type checker should assume this is a non-member attribute
```
class Pet(Enum):
genus: str # Non-member attribute
species: str # Non-member attribute
CAT = 1 # Member attribute
DOG = 2 # Member attribute
```
https://typing.python.org/en/latest/spec/enums.html#defining-members
This prevents the include of the type defined in the pyi,
otherwise this leads to error message like this:
error: Name XYZ already defined (possibly by an import) [no-redef]
* [Python] Use correct type for str with None
Otherwise mypy will correctly flag code like this
def __init__(self):
self.fooBar = None # type: Optional[str]
error: Incompatible types in assignment (expression has type "None", variable has type "str")
* [Python] Make list type optional as they can contain None
Fixes an issue where exports were using incorrect relative paths for
>=3 namespace levels. This is fixed by making the starting range of the
namespace components relative to the amount of components.
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
Adding support for windows requires the code generations
to add a compiler statement to completely ignore GRPC code
generation on windows
Cleanup the project to use the main Package.swift to run tests
instead of having it separate and includes the imports for GRPC
within it.
Adds windows swift ci
* clang-format
* [Python] Replace . with _ in grpc filename suffix
Having filenames with . like `file.fb.grcp`
is not great for Python. Since dots are used for namespaces.
Replacing all of them with _ eg suffix `foo.bar.baz` will become
`foo_bar_baz`.
Restoring the previous default `_fb` suffix.
* [Python] Use namespace in path
This fixes a regression introduced with:
fb9afbafc7
And generates the grpc file in the namespace folder again.
* Sync commandline docs with web docs
* Offical Swift port for FlexBuffers
This is the offical port for FlexBuffers within
swift, and it introcudes a Common Module where code
is shared between flatbuffers and flexbuffers.
Writing most supported values like maps, vectors,
nil and scalars into a flexbuffer buffer. And includes
tests to verify that its similar to cpp
* Reading a flexbuffer
Implementing reading from a flexbuffer, enabling
most of the buffers features, like most types, maps, vectors,
typedvectors, and fixedtypedvectors.
Currently, if an offset/object cant be read we default to a swift
nil instead of the default flexbuffers 'null' with all values.
* Fixes bazel breaking due to new project structure
Address warnings within the library
* Adds comment on why we added the code & properly enforce the amout of bytes needed
For Vector or Array of structures the dereference operator of an
iterator returns the pointer to the structure. However, IndirectHelper,
which is used in the implementation of this operator, is instantiated
in the way that the IndirectHelper::Read returns structure by value.
This is because, Vector and Array instantiate IndirectHelper with
const T*, but VectorIterator instantiates IndirectHelper with T. There
are three IndirectHelper template definition: first for T, second for
Offset<T> and the last one for const T*. Those have different
IndirectHelper:Read implementations and (more importantly) return type.
This is the reason of mismatch in VectorIterator::operator* between
return type declaration and what was exactly returned.
That is, for Array<T,...> where T is scalar the VectorIterator is
instantiated as VectorIterator<T, T>, dereference operator returns T
and its implementation uses IndirectHelper<T> which Read function
returns T.
When T is not scalar, then VectorIterator is instantiated as
VectorIterator<T, const T *>, dereference operator returns const T * and
its implementation uses IndirectHelper<T> which Read function returns T.
The fix is done as follows:
* implement type trait is_specialization_of_Offset and
is_specialization_of_Offset64,
* change partial specialization of IndirectHelper with const T * that
it is instantiated by T and enabled only if T is not scalar and not
specialization of Offset or Offset64,
* remove type differentiation (due to scalar) from Array..
The above makes the IndirectHelper able to correctly instantiate itself
basing only on T. Thus, the instantiation in VectorIterator correctly
instantiate IndirectHelper::Read function, especially the return type.
Fixes access to union members when generating code with options "--cpp-field-case-style upper" and "--gen-object-api"
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
* Adds new API to reduce memory copying within swift
Adds new storage container _InternalByteBuffer which
will be holding the data that will be created within the swift
lib, however reading data will be redirected to ByteBuffer, which
should be able to handle all types of data that swift provide without
the need to copy the data itself. This is due to holding a reference to
the data.
Replaces assumingMemoryBinding with bindMemory which is safer
Adds function that provides access to a UnsafeBufferPointer for
scalars and NativeStructs within swift
Updates docs
Suppress compilation warnings by replacing var with let
Using overflow operators within swift to improve performance
Adds tests for GRPC message creation from a retained _InternalByteBuffer
* flatbuffers Rust reflection: replace num with num-traits
num crate is a wrapper over num-traits and a few other crates, that
reexports the APIs from all of them. We only need num-traits.
Signed-off-by: Marcin Radomski <dextero@google.com>
* Rust reflection: drop dependency on stdint crate
We only use it to get intmax_t for deriving alignment, which is an alias
for `core::ffi::c_long` [1]. We can use that directly instead.
[1] https://docs.rs/stdint/1.0.0/stdint/type.intmax_t.html
Signed-off-by: Marcin Radomski <dextero@google.com>
* Rust reflection: drop dependency on escape_string crate
It's used to format a string used for debugging only, so we might as
well use the builtin Debug representation of a string.
Signed-off-by: Marcin Radomski <dextero@google.com>
* Rust codegen: add derives on generated bitflags
Otherwise it limits the use of structs generated for reflection.fbs
in Rust reflection API.
Signed-off-by: Marcin Radomski <dextero@google.com>
* Rust flatbuffers: update bitflags dependency to 2.8
Signed-off-by: Marcin Radomski <dextero@google.com>
* Rust codegen: use bitflags v2 API for converting from bits
from_bits_unchecked was replaced with safe from_bits_retain.
Signed-off-by: Marcin Radomski <dextero@google.com>
* Regenerate Rust code after idl change
Signed-off-by: Marcin Radomski <dextero@google.com>
* Regenerate reflection_generated.rs
With flatc --rust ../../../reflection/reflection.fbs
Signed-off-by: Marcin Radomski <dextero@google.com>
* ts/BUILD.bazel: add missing import
Found by Buildifire presubmit:
Function "sh_binary" is not global anymore and needs to be loaded from
"@rules_shell//shell:sh_binary.bzl".
Signed-off-by: Marcin Radomski <dextero@google.com>
* Update expected value in generated_code_debug_prints_correctly test
In bitflags v2, the debug string representation of enum values is
different than it was in v1:
Blue -> Color(Blue)
(empty) -> LongEnum(0x0)
This change adjusts the expected test value.
Signed-off-by: Marcin Radomski <dextero@google.com>
* Fix tests build on Swift 5.8
grpc-swift 1.4.1 depends on swift-nio-ssl 2.14.0+ [1]. swift-nio-ssl 2.29.1
published on 2025-01-30, introduced some code [2] that uses a "switch
expression syntax" supported since Swift 5.9 [3]. Attempts to compile it with
Swift 5.8 cause build errors.
swift-nio-ssl project doesn't seem to support Swift 5.8. A commit from
2024-10-29 removes a "deprecated reference to a Swift 5.8 pipeline" [4].
swift-nio-ssl 2.29.0 is the last version that can be compiled with Swift
5.8. This commit pins it to that exact version.
[1] 66e27d7e84/Package.swift (L33)
[2] 3cb4d5ad12 (diff-bc1db1321ff689c2819245dcce1a3080554f0fc13f81b8d326c97e7d42717c8fR54)
[3] https://github.com/swiftlang/swift-evolution/blob/main/proposals/0380-if-switch-expressions.md
[4] 8a6b89d9a4
---------
Signed-off-by: Marcin Radomski <dextero@google.com>
Co-authored-by: Marcin Radomski <dextero@google.com>
This setup is much simpler than calling Bazel from within Bazel
and making sure files and flags are set up correctly.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Add missing file to filegroup for bazel integration tests
Fixup after a9257b6963.
* Align versions in bazel_respository_test_dir with root
* Update XCode version to 15.2
This is the oldest available version.
* Add WORKSPACE.bzlmod
* Add support for Bazel 7
* Add support for Bazel 8 in CI
* Restrict visibility of exported file
* Align npm_translate_lock attrs
* Remove defs_bzl_filename attr
* Align root_package with pnpm-lock.yaml location
Use a symlink to avoid copying the file.
* Update Buildkite Bazel CI
Restructure presubmit.yml to support matrix.
* Remove testing Ubuntu 18.04
The available LLVM 6.0 is too old to support std::filesystem.
* Add testing in Ubuntu 22.04
* Use Bazel version 6.5.0 in integration test
We need to copy the .fbs files into the package used for .bfbs files.
This is necessary as flatc doesn't provide support to specify the full
output file name for an .fbs file in a different folder.
I tried OUTPUT_FILE env var but this doesn't seem to be honored by
flatc.
Transitive headers like array.h have not been available in the runtime_cc target causing the build to fail. Adding all public headers to make sure transitive headers of flatbuffers.h are available.
* #Rust Create a crate for reflection
* #Rust Add a crate for reflection tests and helper to access schema
* #Rust Get root table of a buffer and access field with schema
* #Rust Add 'Struct' struct and corresponding getter
* #Rust Add functions of getting any table/struct field value as integer/float/string
* #Rust Add setters for scalar fields
* #Rust Add setter for string fields
* #Rust Add getter for Table/Vector fields
* #Rust Add buffer verification
* Add a 'SafeBuffer' struct which provides safe methods for reflection
It verifies buffer against schema during construction and provides all the unsafe getters in lib.rs in a safe way
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Improves vectors performance and adds a benchmark to vectors of offsets in swift
Improves performance for all arrays and for loops
Uses a tuple instead of allocating a struct each time we start iterating over fieldloc
Updates benchmark library
* Fixing swift Wasm ci
* Update grpc-core version
io.grpc:grpc-core package in version 1.36.0 contains multiple [CVE's](https://mvnrepository.com/artifact/io.grpc/grpc-core/1.36.0).
Bump grpc-core version to latest 1.68.0 version to mitigate potential vulnerabilities.
* Update grpc version to 1.67.1
grpc was mistakenly released to maven under version 1.68.0 whenever a real release was done for version 1.67.1 [1]. The mistake was fixed later.
[1] https://github.com/grpc/grpc-java/releases
Fix CVE-2022-25647
The package com.google.code.gson:gson before 2.8.9 is vulnerable to Deserialization of Untrusted Data via the writeReplace() method in internal classes, which may lead to denial of service attacks.
Bump up version of the gson package.
https://github.com/advisories/GHSA-4jrv-ppp4-jm57
This allows this type to meet the requirements of e.g.
std::ranges::range, which is necessary for it to work with the
std::span range constructor, or the "non-legacy" constructor for
Chromium's base::span.
Bug: none
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Update NativeObject.swift
Correct the word.
* Update ByteBuffer.swift
Type parameter does not existing, remove it.
* Update ByteBuffer.swift
Correct the word.
In case when flatbuffers are being used along with other project that defines "max" preprocessor macro, the ::max() in FLATBUFFERS_MAX_BUFFER_SIZE and FLATBUFFERS_MAX_64_BUFFER_SIZE is incorrectly being expanded to the macro. Adding parentheses enforces function-like interpretation.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Move `namer.h` and `idl_namer.h` to `include/codegen` so they can be reused from `grpc` dirqectory.
* [gRPC] Update the Python generator to produce typed handlers and Python stubs if requested.
* [gRPC] Document the newly added compiler flags.
This allows enums to be type check with mypy.
They will still behave like ints ->
> IntEnum is the same as Enum,
> but its members are also integers and can be used anywhere
> that an integer can be used.
> If any integer operation is performed with an IntEnum member,
> the resulting value loses its enumeration status.
https://docs.python.org/3/library/enum.html#enum.IntEnum
Only if the --python-typing flag is set.
* [Python] Generate `.pyi` stub files when `--python-typing` is on.
To support this change, the following modifications were made:
- added a new option to disable `numpy` helpers generation;
- added a new flag to control the target Python version:
`--python-version` can be one of the following:
- `0.x.x` – compatible with any Python version;
- `2.x.x` – compatible with Python 2;
- `3.x.x` – compatible with Python 3.
- added codegen utilities for Python;
- added a note that the generated .py file is empty.
* [C++] Update the validator to skip structs in namespaces other than the current one.
Updates copyright from 2023 to 2024 & formats code - updates formatting rules
Updates CI to run with swift 5.8
Adds wasmer & updates command to run carton as a swift plugin
Update bazelci to also accept swift 5.8
Adds swift 5.10 to the test matrix
* [Python] Generate `.pyi` stub files when `--python-typing` is on.
To support this change, the following modifications were made:
- added a new option to disable `numpy` helpers generation;
- added a new flag to control the target Python version:
`--python-version` can be one of the following:
- `0.x.x` – compatible with any Python version;
- `2.x.x` – compatible with Python 2;
- `3.x.x` – compatible with Python 3.
- added codegen utilities for Python;
- added a note that the generated .py file is empty.
* [Python] Update Bazel build rules.
* [Python] Update Bazel build rules.
* [Python] Run buildifier on BUILD.bazel files.
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
This will allow the code to be compiled with `-Wfloat-equal`
as this would result in the folowing warning/error:
vendor/flatbuffers/include/flatbuffers/base.h:465:69:
error: comparing floating point with == or != is unsafe [-Werror,-Wfloat-equal]
template<typename T> inline bool IsTheSameAs(T e, T def) { return e == def; }
But the way it is used in flatbuffers it is ok to compare floating
points with ==.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Fixes Bazel issues for windows and ci
Fetching boringssl within the flatbuffers repository, to patch the issues
of not being able to upgrade to Xcode 14.3 due to buildkite throwing
errors. The patch was inspired by the tenserflow patch
https://github.com/tensorflow/tensorflow/issues/60191#issuecomment-1496073147
Removes references of swift from the windows pipeline for bazel
Sets github actions to use xcode 14.3 for kotlin and sets the macOS
build for intel cpus.
* Update build.yml
Remove comment that is not relevant any longer.
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Update build.yml to ubuntu-24.04
Apparently g++-13 was removed from the ubuntu-22.04 runners.
We also don't have enterprise runners at 24.04 yet, so just use the free ones for now until we get support for those. CI builds might take longer now.
* Update build.yml
Downgrade to g++12 and revert change to using normal runners
* Update build.yml
Go back to ubuntu-24.04 and update both gcc and clang to their latest versions according to [this](https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2404-Readme.md?plain=1#L16-L20).
* Update build.yml
Go back to g++13 for now, as we get some exotic warning in g++14 for newer C++ standards.
* Update build.yml
Fix the other issues with `macos-latest` going to arm: https://github.com/actions/runner-images/tree/main?tab=readme-ov-file#available-images and that Swift wasn't installed in the ubuntu-24.04 by default.
* Update build.yml
Disable Kotlin MacOs CI
The new options are:
- `--grpc-filename-suffix` controls the suffix of the generated files;
- `--grpc-use-system-headers` controls the type of C++ includes generated;
- `--grpc-search-path` controls the directory that contains gRPC runtime;
- `--grpc-additional-header` allows to provide additional dependencies for the generated code.
* [Python] Fix various codegen problems.
This includes:
- escaping keywords happens **after** converting the case:
- currently, `table ClassT` generate `class = Class()` which is invalid Python;
- imports in `one_file` mode use the filename rather than the type name when resolving module names;
- use `filename_suffix` instead of the hardcoded `_generated` one;
- generate empty files if no structs or enums are available. This makes the set of output files more predictable for Bazel.
* [Python] Fix various codegen problems.
This includes:
- escaping keywords happens **after** converting the case:
- currently, `table ClassT` generate `class = Class()` which is invalid Python;
- imports in `one_file` mode use the filename rather than the type name when resolving module names;
- use `filename_suffix` instead of the hardcoded `_generated` one;
- generate empty files if no structs or enums are available. This makes the set of output files more predictable for Bazel.
When flatbuffers is being used from a project that has no use for
JavaScript, users encounter an error similar to the following:
ERROR: Skipping '@com_github_google_flatbuffers//:flatbuffers': error loading package '@com_github_google_flatbuffers//': Unable to find package for @npm//:defs.bzl: The repository '@npm' could not be resolved: Repository '@npm' is not defined.
WARNING: Target pattern parsing failed.
ERROR: error loading package '@com_github_google_flatbuffers//': Unable to find package for @npm//:defs.bzl: The repository '@npm' could not be resolved: Repository '@npm' is not defined.
INFO: Elapsed time: 0.023s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
currently loading: @com_github_google_flatbuffers//
That's not ideal. Users that only care about C++ for example
shouldn't be forced to deal with rules_js and friends.
This patch attempts to fix that by moving the rules_js-specific things
into the `ts` and `tests/ts` directories. This should allow
non-JavaScript projects to ignore rules_js and friends completely.
Here I basically followed the `rules_foo` example from rules_js:
https://github.com/aspect-build/rules_js/tree/main/e2e/rules_foo
The idea is that flatbuffers has its own npm dependencies regardless
of what other projects may have. This means we should not force the
user to import flatbuffers's npm dependencies. The new
`ts/repositories.bzl` file is used by dependents to import
flatbuffers's dependencies. They can still import their own
dependencies. This cleanup allowed me to move all
JavaScript-specific stuff from the top-level directory into
subdirectories.
There should be no changes in this patch in terms of functionality.
It's just a refactor of the rules_js call sites. Users will have to
add a call to the function in `ts/repositories.bzl` in their own
`WORKSPACE` file. They can use
`tests/ts/bazel_repository_test/WORKSPACE` as an example.
Co-authored-by: Derek Bailey <derekbailey@google.com>
If a schema contains a message named e.g. FooT and a message named Foo
while the Object API suffix is T, then two classes with colliding names
will be generated. This scenario will produce a C++ compiler error, but
it's confusing.
This patch moves the error to the compiler, allowing the user to more
readily act to correct the issue.
Co-authored-by: Michael Beardsworth <beardsworth@intrinsic.ai>
Fix "One Definition Rule" violation when using flatbuffers::Verifier with
FLATBUFFERS_TRACK_VERIFIER_BUFFER_SIZE defined in some compilation units
and not defined in other compilation units.
The fix is to make Verifier a template class, with a boolean template
parameter replacing the "#ifdef" conditionals; to rename it as
VerifierTemplate; and then to use "#ifdef" only for a "using" declaration
that defines the original name Verifier an an alias for the instantiated
template. In this way, even if FLATBUFFERS_TRACK_VERIFIER_BUFFER_SIZE is
defined in some compilation units and not in others, as long as clients
only reference flatbuffers::Verifier in .cc files, not header files, there
will be no ODR violation, since the only part whose definition varies is the
"using" declaration, which does not have external linkage.
There is still some possibility of clients creating ODR violations
if the client header files (rather than .cc files) reference
flatbuffers::Verifier. To avoid that, this change also deprecates
FLATBUFFERS_TRACK_VERIFIER_BUFFER_SIZE, and instead introduces
flatbuffers::SizeVerifier as a public name for the template instance with
the boolean parameter set to true, so that clients don't need to define
the macro at all.
* Reproduce the error in a unit test
Reproduces #8200
* Overload KeyCompareWithValue to work for string-like objects
This fixes#8200.
* Extra tests
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
It appears the upgrade to xcode 14.3 broke the macos build on builkite.
The last good build was using xcode 14.2, so go back to this version
until the issue is resolved.
When building with make, it was failing for me because the target grpctext doesn't exist. I strongly assume this was meant to be grpctest.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Add version of push which takes ContiguousBytes
* Ensure overloads aren't ambiguous
* Add version of createVector
* Add version of push which takes ContiguousBytes
* Ensure overloads aren't ambiguous
* Add version of createVector
* Add similar conditional to other use of ContiguousBytes
* Attempt CI fix
* Use memcpy instead of copyMemory
memcpy is faster in tests
* Add testContiguousBytes
* Add benchmarks
* Add version of createVector
* Add benchmarks
* Update push to copy memory
Since we don't care about endianness, we can simply memcpy the array of scalars
* Remove function and benchmarks
Since we don't care about endianness, a FixedWidthInteger version of createVector isn't needed
* Improve naming
* Add doc comment
* Fix C/C++ Create<Type>Direct with sorted vectors
If a struct has a key the vector has to be sorted. To sort the vector
you can't use "const".
* Changes due to code review
* Improve code readability
* Add generate of JSON schema to string to lib
* option indent_step is supported
* Remove unused variables
* Fix break in test
* Fix style to be consistent with rest of the code
* [TS] Fix reserved words as arguments (#6955)
* [TS] Fix generation of reserved words in object api (#7106)
* [TS] Fix generation of object api
* [TS] Fix MakeCamel -> ConvertCase
* [C#] Fix collision of field name and type name
* [TS] Add test for struct of struct of struct
* Update generated files
* Add missing files
* [TS] Fix query of null/undefined fields in object api
* Generate only files for comiled fbs (not for dependend ones)
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Update build.yml
Use our enterprise runners
* Make a default runs-on
* Update build.yml
Use the latest 64-core runners
* Update build.yml
Fix windows runner that don't have visual studios
* Update build.yml
use windows-2019 as the 2022 doesn't seem to have visual studios installed
* Update build.yml
Upgrade to gcc 13 and clang 15
* switch to __is_trivially_copyable
* fix cmake issue and warning about sign comparison
* Use libc++ for C++23 on clang for now
* Use libc++ for C++23 on clang for now
* exclude clang+15 for C++13 builds
Addresses a warning on xcode 15 regarding copying a pointer without safeguards
Address PR comments regarding initializing buffers with flag
Adds a test case for copying unaligned buffers
Formatting code
`buildifier` was complaining as follows:
#### :bazel: buildifier: found 2 lint issues in your WORKSPACE, BUILD and *.bzl files
<pre><code>tests/ts/bazel_repository_test_dir/BUILD:3:1: <a href="https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#out-of-order-load">out-of-order-load</a>: Load statement is out of its lexicographical order.
ts/BUILD.bazel:2:1: <a href="https://github.com/bazelbuild/buildtools/blob/master/WARNINGS.md#out-of-order-load">out-of-order-load</a>: Load statement is out of its lexicographical order.</pre></code>
This can be fixed locally like so:
$ buildifier -lint fix $(git ls-files | grep -e '/BUILD.bazel$' -e '/BUILD$' -e '\<WORKSPACE$')
I also took this opportunity to fix one of the filenames.
I accidentally introduced these errors in #8078.
If you used flatbuffers_ts_library with gen_reflections = True then it
attempted to use the flat-file compiler rather than flatc itself.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* [Kotlin] Small optimizations and benchmark on deserialization
* [Kotlin] Remove redudant assign() method (use init() instead)
* [Kotlin] Fix benchmark run after change in flatbuffers-java deps
Commit 6e214c3a49 fixes Kotlin build,
but makes the kotlin-benchmark plugin misses the java classes at
runtime, causing NotClassFoundError. The alternative to solve the issue
is to read java's pom.xml to get the latest java version and use it
as dependency. With that we avoid compilation errors on a new version and
keep benchmark plugin happy.
By using specific jar version to use java's runtime on the benchmark
module we let CI break when new versions are released. So we are using
source directly instead
* [Kotlin] Introduction to Kotlin Multiplaform
The first implementation of the Kotlin code generation was made years
ago at the time Kotlin Multiplaform was not stable and Kotlin is mostly
used on JVM-based targets. For this reason the generated code uses java
based runtime.
That design decision comes with many drawbacks, leaving the code
generated more java-like and making it impossible to use more advanced
features of the Kotlin language.
In this change we are adding two parts: A pure, multi-plaform, Kotlin
runtime and a new code generator to accompany it.
* [Kotlin] Remove scalar sign cast from code generation
Now that we have a new runtime the accepts unsigned types, we don't
need to code generate casting back and from signed scalars. This
MR removes this from both code generations and adds the necessary
API to the runtime.
* [Kotlin] Use offset on public API to represent buffer position
Currently, kotlin was following Java's approach of representing objects,
vectors, tables as "Int" (the position of it in the buffer). This change
replaces naked Int with Offset<T>, offering a type-safe API. So,
instead of
fun Table.createTable(b: FlatBufferBuilder, subTable: Int)
We will have
fun Table.createTable(b: FlatBufferBuilder, subTable: Offset<SubTable>)
Making impossible to accidentally switch parameters.
The performance should be similar to use Int as we are using value
class for Offset and ArrayOffset, which most of the time translate to
Int in the bytecode.
* [Kotlin] Add builder for tables
Add builder constructor to make create of table more ergonomic.
For example the movie sample for the test set could be written as:
Movie.createMovie(fbb,
mainCharacterType = Character_.MuLan,
mainCharacter = att) {
charactersType = charsType
this.characters = characters
}
instead of:
Movie.startMovie(fbb)
Movie.addMainCharacterType(fbb, Character_.MuLan)
Movie.addMainCharacter(fbb, att as Offset<Any>)
Movie.addCharactersType(fbb, charsType)
Movie.addCharacters(fbb, charsVec)
Movie.endMovie(fbb)
* [Kotlin] Move enum types to value class
Moving to flatbuffer enums to value class adds type safety for parameters
with minimum to no performance impact.
* [Kotlin] Simplify Union parameters to avoid naked casting
Just a small change on the APIs that receive union as parameters,
creating a typealias UnionOffset to avoid using Offset<Any>. To "convert"
an table offset to an union, one just call Offset.toUnion().
* [Kotlin] Apply clang-format on kotlin code generators
* [Kotlin] Update kotlin generator to follow official naming conventions
Updating directory, package and enum naming to follow Kotlin official
convention.
https://kotlinlang.org/docs/coding-conventions.html#naming-rules
* [Kotlin] Add fixes to improve performance
1 - Add benchmark comparing serialization between Java & Kotlin
2 - ReadWriteBuffer does not auto-grow (thus avoid check size in every op)
3 - Add specialized add functions on FlatBufferBuilder to avoid boxing
offsets.
4 - Remove a few Kotlin syntax sugar that generated performance penalties.
* [Kotlin] Remove builder from Kotlin KMP and add some optimizations
to avoid boxing of Offset classes
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
The test was not actually invoking the bazel that was downloaded with
the `http_file` rule. I failed to add `executable = True` to the
`http_file` call. This caused the test to ignore that bazel binary and
went to the next one on the system.
This patch fixes the issue by adding the missing attribute. Also, this
patch changes the check in the test to make sure that the downloaded
file is indeed executable.
* First working hack of adding 64-bit. Don't judge :)
* Made vector_downward work on 64 bit types
* vector_downward uses size_t, added offset64 to reflection
* cleaned up adding offset64 in parser
* Add C++ testing skeleton for 64-bit
* working test for CreateVector64
* working >2 GiB buffers
* support for large strings
* simplified CreateString<> to just provide the offset type
* generalize CreateVector template
* update test_64.afb due to upstream format change
* Added Vector64 type, which is just an alias for vector ATM
* Switch to Offset64 for Vector64
* Update for reflection bfbs output change
* Starting to add support for vector64 type in C++
* made a generic CreateVector that can handle different offsets and vector types
* Support for 32-vector with 64-addressing
* Vector64 basic builder + tests working
* basic support for json vector64 support
* renamed fields in test_64bit.fbs to better reflect their use
* working C++ vector64 builder
* Apply --annotate-sparse-vector to 64-bit tests
* Enable Vector64 for --annotate-sparse-vectors
* Merged from upstream
* Add `near_string` field for testing 32-bit offsets alongside
* keep track of where the 32-bit and 64-bit regions are for flatbufferbuilder
* move template<> outside class body for GCC
* update run.sh to build and run tests
* basic assertion for adding 64-bit offset at the wrong time
* started to separate `FlatBufferBuilder` into two classes, 1 64-bit aware, the other not
* add test for nested flatbuffer vector64, fix bug in alignment of big vectors
* fixed CreateDirect method by iterating by Offset64 first
* internal refactoring of flatbufferbuilder
* block not supported languages in the parser from using 64-bit
* evolution tests for adding a vector64 field
* conformity tests for adding/removing offset64 attributes
* ensure test is for a big buffer
* add parser error tests for `offset64` and `vector64` attributes
* add missing static that GCC only complains about
* remove stdint-uintn.h header that gets automatically added
* move 64-bit CalculateOffset internal
* fixed return size of EndVector
* various fixes on windows
* add SizeT to vector_downward
* minimze range of size changes in vector and builder
* reworked how tracking if 64-offsets are added
* Add ReturnT to EndVector
* small cleanups
* remove need for second Array definition
* combine IndirectHelpers into one definition
* started support for vector of struct
* Support for 32/64-vectors of structs + Offset64
* small cleanups
* add verification for vector64
* add sized prefix for 64-bit buffers
* add fuzzer for 64-bit
* add example of adding many vectors using a wrapper table
* run the new -bfbs-gen-embed logic on the 64-bit tests
* remove run.sh and fix cmakelist issue
* fixed bazel rules
* fixed some PR comments
* add 64-bit tests to cmakelist
* Add binary schema reflection
* remove not-used parameter
* move logic from object API to base API
* forward declare
* remove duplicate code gen that was stompping on the edits
* reduce to just typedef generation
* fixed bazel rules to not stomp
* more bazel fixes to support additional generated files
* Migrate from rules_nodejs to rules_js/rules_ts (take 2)
This is the second version of patch #7923. The first version got
reverted because bazel query was failing:
$ bazel --nosystem_rc --nohome_rc query tests(set('//...')) except tests(attr("tags", "manual", set('//...')))
ERROR: Traceback (most recent call last):
File "/workdir/tests/ts/bazel_repository_test_dir/BUILD", line 6, column 22, in <toplevel>
npm_link_all_packages(name = "node_modules")
File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/external/npm/defs.bzl", line 188, column 13, in npm_link_all_packages
fail(msg)
Error in fail: The npm_link_all_packages() macro loaded from @npm//:defs.bzl and called in bazel package 'tests/ts/bazel_repository_test_dir' may only be called in bazel packages that correspond to the pnpm root package '' and pnpm workspace projects ''
This was happening because the `.bazelrc` file only added
`--deleted_packages` to the `build` command. We also need it for the
`query` command. This second version of the patch fixes that.
Original commit message:
This patch migrates the current use of rules_nodejs to the new rules_js.
rules_js is the intended replacement of rules_nodejs as per this note:
https://github.com/aspect-build/rules_js#relationship-to-rules_nodejs
> rules_js is an alternative to the build_bazel_rules_nodejs Bazel module
> and accompanying npm packages hosted in
> https://github.com/bazelbuild/rules_nodejs, which is now
> unmaintained. All users are recommended to use rules_js instead.
There are a few notable changes in this patch:
1. The `flatbuffer_ts_library` macro no longer accepts a `package_name`
attribute. This is because rules_js appears to manage the import
naming of dependencies via top-level `npm_link_package` targets.
Users will have to migrate.
2. I added a few more arguments to `flatbuffer_library_public()`. These
helped with exposing esbuild to `ts/compile_flat_file.sh`.
3. I pinned the version of `typescript` in `package.json` so that
rules_ts can download the exact same version. rules_ts doesn't know
what to do if the version isn't exact.
4. Since rules_js uses the pnpm locking mechanism, we now have a
`pnpm-lock.yaml` file instead of a yarn lock file.
4. I added bazel targets for a few of the existing tests in `tests/ts`.
They can be run with `bazel test //test/ts:all`. Since there is no
flexbuffers bazel target, I did not add a bazel target for the
corresponding test.
5. I added a separate workspace in `tests/ts/bazel_repository_test_dir/`
to validate that the flatbuffers code can be imported as an external
repository. You can run the test with
`bazel test //test/ts:bazel_repository_test`. For this to work, I
needed to expose a non-trivial chunk of the flatbuffers code to the
test. I achieved this through some recursive `distribution`
filegroups. This is inspired by rules_python's workspace tests.
I did not do anything special to validate that the `gen_reflections`
parameter works the same. This patch doesn't change anything about
the TypeScript generation.
As a side note: I am not an expert with rules_js. This patch is my
attempt based on my limited understanding of the rule set.
Fixes#7817
* Fix the query
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Start using pnpm
* Add @npm
* get more stuff set up
* Get the analysis phase passing.
* Get esbuild working?
* Get it compiling?
$ bazel build //tests/ts/...
* Try to get the test working
* test is passing
* Get the other tests working
* clarify comment
* clean up a bit
* Try to add another test
* Add another test
* clean up more
* remove unused reference
* Add e2e test
* Get more of the test working
* add lock file
* Get test working on its own
* Get e2e test passing
* fix infinite recursion
* Add comments
* clean up some more
* clean up more again
* Source typescript version from package.json
* run buildifier
* lint
* Fix unset `extra_env`
* Incorporate feedback
* run buildifier
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
absl::string_view is uses std::string_view when available. It already checks if std::string_view is available in the earlier code.
It should only use absl::string_view implementation.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* optionally generate type prefixes and suffixes for python code
* fix codegen error when qualified name is empty
* WIP: Python typing
* more progress towards python typing
* Further iterate on Python generated code typing
* clang-format
* Regenerate code
* add documentation for Python type annotations option
* generate code with Python type annotations
* handle forward references
* clang-format
This change allows user to decode binary with given schema to JSON
representation when schema defines union with struct.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Fix C/C++ Create<Type>Direct with sorted vectors
If a struct has a key the vector has to be sorted. To sort the vector
you can't use "const".
* Changes due to code review
* Improve code readability
* Add generate of JSON schema to string to lib
* option indent_step is supported
* Remove unused variables
* Fix break in test
* Fix style to be consistent with rest of the code
* [TS] Fix reserved words as arguments (#6955)
* [TS] Fix generation of reserved words in object api (#7106)
* [TS] Fix generation of object api
* [TS] Fix MakeCamel -> ConvertCase
* [C#] Fix collision of field name and type name
* [TS] Add test for struct of struct of struct
* Update generated files
* Add missing files
* [TS] Fix query of null/undefined fields in object api
* Add .Net verfier
* Add some fuzz tests for .Net
* Remove additional files
* Fix .net test
* Changes due to PR
* Fix generated files
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
PyPI has three capital letters. See the front page of the service:
https://pypi.org/
"The Python Package Index (PyPI) ..."
Update the Python link under "Supported programming languages"
Co-authored-by: Michael Le <michael.le647@gmail.com>
ToCamelCase(input, true) converts first char to upper case, but
ToCamelCase(input, false) keeps the case of the first char. We are
changing its behavior to force a lower case.
Co-authored-by: Derek Bailey <derekbailey@google.com>
Shorten the PR staleness from 6 months to 3 weeks + 1 week notice. PRs become much harder to deal with the old they become due to merge conflicts and divergence.
Updated to stale@v7.0.0
The distributions for C++ and Python include the generated reflection
bindings but are currently missing from the other language packages.
This will bring the Java package generated for releases closer to
feature parity with the C++ and Python release artifacts.
* TS/JS: Export object based classes on entry
Along with the non object ones, for consistency. This is a regression
introduced recently.
Before:
`export { UpdateSettingsRequest } from './worker/update-settings-request.js';`
Now:
`export { UpdateSettingsRequest, UpdateSettingsRequestT } from './worker/update-settings-request.js';`
* only export object based classes for structs
Enums are not elegible.
---------
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
Co-authored-by: Derek Bailey <derekbailey@google.com>
* [Android] fixed build after decomission of jcenter
JCenter[1] has been removed and now is failing android build. This
change updates the configuration to remove this and few other warnings.
1 - https://developer.android.com/studio/build/jcenter-migration
* [Kotlin] fix build for latest gradle version 8.0.1
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
The generation of the library interface supplied by this function only
works within the same directory as that the target was defined. By
adding a custom target named GENERATE_<TARGET> now also interface files
will be generated by making a target dependend on the generate target.
Example:
/CMakeLists.txt
set(MY_INCL_SRC ${CMAKE_CURRENT_SOURCE_DIR}/fbs/my_incl.fbs)
flatbuffers_generate_headers(TARGET my_incl
SCHEMAS ${MY_INCL_SRC})
add_subdirectory(app)
/app/CMakeLists.txt
add_executable(app src/test.cpp)
target_link_libraries(app my_incl)
add_dependencies(app GENERATE_my_incl)
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Fix C/C++ Create<Type>Direct with sorted vectors
If a struct has a key the vector has to be sorted. To sort the vector
you can't use "const".
* Changes due to code review
* Improve code readability
* Add generate of JSON schema to string to lib
* option indent_step is supported
* Remove unused variables
* Fix break in test
* Fix style to be consistent with rest of the code
* [TS] Fix reserved words as arguments (#6955)
* [TS] Fix generation of reserved words in object api (#7106)
* [TS] Fix generation of object api
* [TS] Fix MakeCamel -> ConvertCase
* [C#] Fix collision of field name and type name
* [TS] Add test for struct of struct of struct
* Update generated files
* Add missing files
* [TS] Fix query of null/undefined fields in object api
* Fix collision if field name is equal to table name and used as key in an array
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Add code generator for proto files
* Update
* Add --proto to script
* Remove cmt
* Move proto parsing logic into else block to share same set up logic for code_generator
* Remove IsValidCodeGenerator
If flatbuffers is built in C++11 mode, but there is a recent version of
absl which requires C++14, the build will fail.
Cf https://github.com/MapServer/MapServer/issues/6822 for the use case
that triggered this.
* Parsing from proto should keep field ID. (fixes#7645)
* Fix failed tests
* Fix windows warning
* Improve attribute generation in proto to fbs
* Check if id is used twice. fix Some clang-format problems
* Test if fake id can solve the test problem
* Validate proto file in proto -> fbs generation.
* Fix error messages
* Ignore id in union
* Add keep proto id for legacy and check gap flag have been added. Reserved id will be checked.
* Add needed flags
* unit tests
* fix fromat problem. fix comments and error messages.
* clear
* More unit tests
* Fix windows build
* Fix include problems
* Fake commit to invoke rebuild
* Fix buzel build
* Fix some issues
* Fix comments, fix return value and sort for android NDK
* Fix return type
* Break down big function
* Place todo
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
The current detection method fails on GCC 12.2 with -std=c++20 because
the __cpp_lib_span macro is undefined.
As per https://en.cppreference.com/w/cpp/utility/feature_test ,
__cpp_lib_span requires including either <version> or <span>.
Since both these headers were added in C++20, checking for C++20 is
sufficient (and simpler than using the library feature-test macro).
Signed-off-by: Bernie Innocenti <bernie@codewiz.org>
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Refactor to use CodeGenerator interface.
- Move code to its own header file to be included in flatc_main.cpp
- Refactor code to use CodeGenerator interface for all languages
* Format all files
* remove lua code generator since it doesn't support bfbs generator
* Update CMakeLists file with new idl_gen_*.cpp and idl_gen_*.h files
* Add idl_gen_swift header file
* Add idl_gen_swift header file and update bazel file
* Remove CodeGenerator interface for idl_gen_text.*. Remove comments and extern declaration
* Reorder header and implementation files in CMakeLists.txt
* Add idl_gen_* header files to implementation files
* Update CMakeLists and remove unused import
Co-authored-by: Derek Bailey <derekbailey@google.com>
* add unit tests for support struct as key
* make changes to parser and add helper function to generate comparator for struct
* implement
* add more unit tests
* format
* just a test
* test done
* rerun generator
* restore build file
* address comment
* format
* rebase
* rebase
* add more unit tests
* rerun generator
* address some comments
* address comment
* update
* format
* address comment
Co-authored-by: Wen Sun <sunwen@google.com>
Co-authored-by: Derek Bailey <derekbailey@google.com>
* [TS/JS] Entry point per namespace
* Fix handling of outputpath and array_test
* Attempt to fix generate_code
* Fix cwd for ts in generate_code
* Attempt to fixup bazel and some docs
* Add --ts-flat-files to bazel build to get bundle
* Move to DEFAULT_FLATC_TS_ARGS
* Attempt to add esbuild
* Attempt to use npm instead
* Remove futile attempt to add esbuild
* Attempt to as bazel esbuild
* Shuffle
* Upgrade bazel deps
* Revert failed attempts to get bazel working
* Ignore flatc tests for now
* Add esbuild dependency
* `package.json` Include esbuild
* `WORKSPACE` Add fetching esbuild binary
* Update WORKSPACE
* Unfreeze Lockfile
* Update WORKSPACE
* Update BUILD.bazel
* Rework to suggest instead of running external bundler
* Add esbuild generation to test script
* Prelim bundle test
* Run test JavaScriptTest from flatbuffers 1.x
* Deps upgrade
* Clang format fix
* Revert bazel changes
* Fix newline
* Generate with type declarations
* Handle "empty" root namespace
* Adjust tests for typescript_keywords.ts
* Separate test procedure for old node resolution module output
* Fix rel path for root level re-exports
* Bazel support for esbuild-based flatc
Unfortunately, we lose typing information because the new esbuild method
of generating single files does not generate type information.
The method used here is a bit hack-ish because it relies on parsing the
console output of flatc to figure out what to do.
* Try to fix bazel build for when node isn't present on host
* Auto formatting fixes
* Fix missing generated code
Co-authored-by: Derek Bailey <derekbailey@google.com>
Co-authored-by: James Kuszmaul <jabukuszmaul+collab@gmail.com>
* Fix binary output different in different platform, due to the nan serialization
* Add check generated code on windows ci
* Remove resdundant script
* Fix eof, and check script
* Minor bug in gen code script
* Fix windows script, remove redundant scripts
* Undelete redundante codes
* Fix github action
* Ignore eof generate grpc
Co-authored-by: Derek Bailey <derekbailey@google.com>
Comparing short strings, small integers, and Booleans by identity
(memory address) can work due to optimizations in the Python
interpreter, but it is neither formally correct nor reliable. Use
equality comparisons instead.
Co-authored-by: Derek Bailey <derekbailey@google.com>
It is deprecated in favour of importlib and slated for removal in Python
3.12. Since the return value of imp.find_module('numpy') is unused, the
only effect of calling this function is to raise an ImportError when
numpy is not available; importing numpy directly is already sufficient
to do this.
The imp package is still used in python/flatbuffers/compat.py, but only
on Python 2, where it is not deprecated and will not be removed.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* [TS]: Fix vtable creation for consecutive required fileds (#7739)
* handle feedback
* comment the schema
* comment change in builder.ts
* [TS]: builder, Fix requiredField()
Verifty that the field is present in the vtable.
* restore monsterdata binary file
Co-authored-by: Derek Bailey <derekbailey@google.com>
* [Kotlin] Control the generation of reflection with --reflect-names.
Tested:
```
$ cmake -G "Unix Makefiles" && make && ./tests/flatc/main.py
...
KotlinTests.EnumValAttributes
[PASSED]
KotlinTests.EnumValAttributes_ReflectNames
[PASSED]
KotlinTests: 2 of 2 passsed
...
35 of 35 tests passed
```
* [Kotlin] Fix SampleBinary by converting Byte to UByte for ubyte fields.
* [Kotlin] Annotate all generated classes with kotlin.ExperimentalUnsignedTypes.
Since flatbuffers is using calendar versioning and does not provide
any ABI stability guarantees, use the complete version as SOVERSION
for the shared library rather than just the major component. This
prevents breaking reverse dependencies on incompatible upgrades.
Fixes#7759
* Fix [C#] Object API - Invalid Property Name used in UnPackTo for union fieldhttps://github.com/google/flatbuffers/issues/7750, also fixes invalid Code generated in WriteJson for Unions named Value.
* Test added: new schema union_value_collision.fbs with a Union named Value and a union field named value. The generated C# code now compiles when NetTest.bat. The Code generated with an older flatc.exe didn't compile because of a mismatch of the property name (Value vs. Value_).
* branch was not up-to-date with master
* BASE_OPTS + CPP_OPTS removed and union_value_collision_generated.h deleted
Co-authored-by: Derek Bailey <derekbailey@google.com>
Add the --no-minmax-values flag to prevent flatc from generating C++
enums with MIN and MAX enumerated values that otherwise would be set
to the inclusive lower and upper bound respectively of the enum.
This command-line flag is needed to avoid collisions when an enum that
is being ported to FlatBuffers already has a MIN or MAX enumerated
value.
It is also needed to work around a long-standing problem with
magic_enum that causes magic_enum to not see enumerated values that
are not unique. For example, if FlatBuffers sets MIN = FOO and MAX =
BAR, MIN and FOO share the same underlying value so they are not
unique. The same is true of MAX and BAR. This prevents magic_enum
from converting FOO and BAR to and from strings as well as causing
magic_enum to return a count of enumerated values that is two fewer
than it should be.
Co-authored-by: Paul Serice <paul@serice.net>
* [Kotlin] Only generate nullable return types if the field is not required
* [Kotlin] Fix generated code formatting according to kotlin style guide
Co-authored-by: Derek Bailey <derekbailey@google.com>
Co-authored-by: Paulo Pinheiro <paulovictor.pinheiro@gmail.com>
To make it simple to map between a union field and its union type
field we are adding a pointer to FieldDef to point to each other. For
all other types the pointer will be nullptr.
Co-authored-by: Derek Bailey <derekbailey@google.com>
#7451 caused getFullyQualifiedName to return a name with underscores,
not periods. Because the fully qualified name is a property of
FlatBuffers, not the language being codegen'd for, it should use
periods. Fixes#7564.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Add clang-tidy, fix some bugpron problems.
* Fix more issues
* Fix some more issues :))
* Minimal pr to just add clang-tidy
Co-authored-by: Derek Bailey <derekbailey@google.com>
* create job to build linux and run unit test on s390x
* update
* update
* update
* update
* update
* print out machine type
* create regression test to build a big endian arch and run unit tests daily
* rename and schedule run on pr merged and on request
* udpate
Co-authored-by: Wen Sun <sunwen@google.com>
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Add Ref.AsStringBytes to flatbuffers.flexbuffers Python API
* Append Bytes to AsStringBytes return value
Co-authored-by: Jared Junyoung Lim <jaredlim@google.com>
* update unit test and generated file to test is extra endianswap can help resolve issue
* remove EndianScalar wrapper from Get method
* remove endianscalar wrapper
* update
* update
* use Array instead
* clang format
* address error
* clang
* update
* manually generate
* Move Nim to completed language
* Add swift link
* address comments
* update unit test
* address comment
* address comment
* regenerate file
* use auto instead of size_t
* use uint32_t instead
* update
* format
* delete extra whitespace
Co-authored-by: Wen Sun <sunwen@google.com>
Co-authored-by: Derek Bailey <derekbailey@google.com>
Without the change build fails on weekly `gcc-13` snapshots as:
In file included from /build/flatbuffers/tests/reflection_test.cpp:1:
tests/reflection_test.h:9:57: error: 'uint8_t' has not been declared
9 | void ReflectionTest(const std::string& tests_data_path, uint8_t *flatbuf, size_t length);
| ^~~~~~~
* Update release script to update Rust version (it still needs to be published after)
* Also update rust while I'm at it
Co-authored-by: Casper Neo <cneo@google.com>
* Add --go-module-name flag to support generating code for go modules
* Rename echo example folder
* Grammar
* Update readme for go-echo example
* Update readme for go-echo example
* Re-enable go modules after test is done
* Add support for key lookup for tables in Go
* Run clang format
* Run go fmt on tests
* Remove TODO in tests
* Update LookupByKey API
* Update LookupByKey API
* Don't use resolvePointer in expectEq
* Use generated getters instead of reading values directly from buffer
* Fix typo
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Fix go generator undefined Package, also throw exception in specific examples.
* Add test for go generator import problem
* Add new version of generated go file. Fix conflict.
* Add executable permission to generate_code.py script.
* Improve test quality, remove unwanted generated files, better naming
* Fix comments
* clang format
Co-authored-by: Derek Bailey <derekbailey@google.com>
The BytesConsumed function uses the `cursor_` to determine how many
bytes have been consumed by the parser, in case the user of the Parser
object wants to step over the parsed flatbuffer that is embedded in some
larger string. However, the `cursor_` is always one token ahead, so that
it can determine how to consume it. It points at the token that is about
to be consumed, which is ahead of the last byte consumed.
For example, if you had a string containing these two json objects and
parsed them...
"{\"key\":\"value\"},{\"key\":\"value\"}"
...then the `cursor_` would be pointing at the comma between the two
tables. If you were to hold a pointer to the beginning of the string and
add `BytesConsumed()` to it like so:
const char* json = // ...
parser.ParseJson(json);
json += parser.BytesConsumed();
then the pointer would skip over the comma, which is not the expected
behavior. It should only consume the table itself.
The solution is simple: Just hold onto a previous cursor location and
use that for the `BytesConsumed()` call. The previous cursor location
just needs to be set to the cursor_ location each time the cursor_ is
about to be updated. This will result in `BytesConsumed()` returning
the correct number of bytes without the off-by-one-token error.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Fix C/C++ Create<Type>Direct with sorted vectors
If a struct has a key the vector has to be sorted. To sort the vector
you can't use "const".
* Changes due to code review
* Improve code readability
* Add generate of JSON schema to string to lib
* option indent_step is supported
* Remove unused variables
* Fix break in test
* Fix style to be consistent with rest of the code
* [TS] Fix reserved words as arguments (#6955)
* [TS] Fix generation of reserved words in object api (#7106)
* [TS] Fix generation of object api
* [TS] Fix MakeCamel -> ConvertCase
* [C#] Fix collision of field name and type name
* [TS] Add test for struct of struct of struct
* Update generated files
* Add missing files
* [TS] Fix query of null/undefined fields in object api
* [C#] Fix collision of member if enum name is "Value"
* Fix due to style guide
Co-authored-by: Derek Bailey <derekbailey@google.com>
* add support for using array of scalar as key field
* update cmakelist and test.cpp to include the tests
* update bazel rule
* address comments
* clang format
* delete comment
* delete comment
* address the rest of the commnets
* address comments
* update naming in test file
* format build file
* buildifier
* make keycomparelessthan call keycomparewithvalue
* update to use flatbuffer array instead of raw pointer
* clang
* format
* revert format
* revert format
* update
* run generate_code.py
* run code generator
* revert changes by generate_code.py
* fist run make flatc and then run generate_code.py
Co-authored-by: Wen Sun <sunwen@google.com>
* Fix C/C++ Create<Type>Direct with sorted vectors
If a struct has a key the vector has to be sorted. To sort the vector
you can't use "const".
* Changes due to code review
* Improve code readability
* Add generate of JSON schema to string to lib
* option indent_step is supported
* Remove unused variables
* Fix break in test
* Fix style to be consistent with rest of the code
* [TS] Fix reserved words as arguments (#6955)
* [TS] Fix generation of reserved words in object api (#7106)
* [TS] Fix generation of object api
* [TS] Fix MakeCamel -> ConvertCase
* [C#] Fix collision of field name and type name
* [TS] Add test for struct of struct of struct
* Update generated files
* Add missing files
* [TS] Fix query of null/undefined fields in object api
* Put documentation to bfbs if it is not empty
* Fix monster test bfbs reference files
* Fix generated monster test files
Why they are different when generating it with linux and windows executable?
* Fix import problem in dart generated files. (fixes#7609).
* Fix naming.
* Fix minor changes in generated files.
* Add some tests. Fix minor problems.
* Fix minor format problem plus import alias issue.
* Minor fix in dart code generator, remove java from examples
* remove java and go generated files
* Fix dart tests.
* Fix spell problem.
* Remove excessive tests :))
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Temporary fix for SLSA generators
Sigstore made a breaking change as part of their recent GA announcement. We need a temporary fix to avoid builder failure (see slsa-framework/slsa-github-generator#1163)
/cc @asraa
* Update build.yml
+/-inf were not being handled, and so invalid typescript was being
generated when a float/double had an infinite default value. NaN was
being handled correctly.
Co-authored-by: Derek Bailey <derekbailey@google.com>
Co-authored-by: Casper <casperneo@uchicago.edu>
* Add support for proto 3 map to fbs gen
* Run clang-format
* Update proto golden test
* Rename variables
* Remove iostream
* Remove iostream
* Run clang format
Co-authored-by: Derek Bailey <derekbailey@google.com>
Previously when parsing a JSON representation of a Flatbuffer, the
parser required that the input string contain one and only one root
table. This change adds a flag that removes that requirement, so that
if a Flatbuffer table is embedded in some larger string the parser will
simply stop parsing once it reaches the end of the root table, and does
not validate that it has reached the end of the string.
This change also adds a BytesConsumed function, which returns the number
of bytes the parser consumed. This is useful if the table embedded in
some larger string that is being parsed, and that outer parser needs to
know how many bytes the table was so that it can step over it.
* [TS] Add support for fixed length arrays on Typescript (#5864) (#7021)
* Typescript / Javascript don't have fixed arrays but it is important to support these languages for compatibility.
* Generated TS code checks the length of the given array and do truncating / padding to conform to the schema.
* Supports the both standard API and Object Based API.
* Added a test.
Co-authored-by: Mehmet Baker <mehmet.baker@zerodensity.tv>
Signed-off-by: Bulent Vural <bulent.vural@zerodensity.tv>
Signed-off-by: Bülent Vural <bulent.vural@zerodensity.tv>
* Formatting & readability fixes on idl_gen_ts.cpp
Signed-off-by: Bülent Vural <bulent.vural@zerodensity.tv>
* Added array_test_complex.bfbs
Signed-off-by: Bülent Vural <bulent.vural@zerodensity.tv>
* TS arrays_test_complex: Remove bfbs and use fbs directly
Signed-off-by: Bülent Vural <bulent.vural@zerodensity.tv>
Signed-off-by: Bülent Vural <bulent.vural@zerodensity.tv>
* feat: Fixed the issue with nested unions relying on InitFromBuf.
Problem: Issue #7569
Nested Unions were broken with the introduction of parsing buffers with an initial encoding offset.
Fix:
Revert the InitFromBuf method to the previous version and introduction of InitFromPackedBuf that allows
users to read types from packed buffers applying the offset automatically.
Test:
Added in TestNestedUnionTables to test the encoding and decoding ability using a nested table with a
union field.
* fix: Uncommented generate code command
Create a new `release.yml` for defining automatic publishing to package managers on releases.
Adds the NPM publishing steps as it is the most straightforward.
* Bfbs Nim Generator
* Remove commented out tests
* add missing line to idl.h
* Commit python reflection changes
* Commit python reflection changes and move tests
* Remove default string addition
* Move tests to python file
* Fix element size check when element is table
* remove whitespace changes
* add element_type docs and commit further to namer and remove kkeep
* Bfbs Nim Generator
* Remove commented out tests
* add missing line to idl.h
* Commit python reflection changes
* Commit python reflection changes and move tests
* Remove default string addition
* Move tests to python file
* Fix element size check when element is table
* remove whitespace changes
* add element_type docs and commit further to namer and remove kkeep
* remove unused variables
* added tests to ci
* added tests to ci
* fixes
* Added reflection type Field, Variable to namer
* Moved reflection namer impl to bfbsnamer
* Remove whitespace at end of line
* Added nim to generated code
* Revert whitespace removal
Co-authored-by: Derek Bailey <derekbailey@google.com>
As Java does not support unsigned integer types, the value types
are "rounded up" (an uint32 is represented as a long) but persisted
correctly (an uint32 is persisted as 4 bytes).
This CL makes a cast operation explicit so that the compiler
does not throw warning messages.
Co-authored-by: Dominic Battre <battre@chromium.org>
Co-authored-by: Derek Bailey <derekbailey@google.com>
The MyGame/Example/LongEnum.java class did not compile because
Java expects an "L" suffix for literals of type long.
This CL fixes the code generation to include such a suffix.
Co-authored-by: Dominic Battre <battre@chromium.org>
on a subset of platforms. The test calls are guarded via #ifndef
FLATBUFFERS_NO_FILE_TEST.
Embedders of flatbuffers that rely on -Werror,-Wunused-function
compiler flags (like chromium) complain about the exsitence of these
tests in the anonymous namespace.
This CL guards the test definitions as well (not just the test
calls).
Co-authored-by: Dominic Battre <battre@chromium.org>
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Add support for metadata attributes for enum values (#7567)
* Fix path lookup in flatc test
* Try a fix for Windows paths
* Convert path to string to fix Windows error
* feat: Added support for fixed sized arrays to python
Problem:
We encountered that using fixed arrays from C++ to python that python would
not read those arrays correctly due to no size information being encoded in the byte
array itself.
Fix:
Encode the sizes within the generated python file during code generation.
Specfically we add GetArrayAsNumpy to the python version of table, which takes as input
the length of the vector. When generating the python message files we include this length
from the VectorType().fixed_length.
* fix: added digit support for camel case to snake case conversion
Problem:
When including a number in the message name we would encounter cases where SnakeCase would
not add the appropirate breaks. e.g. Int32Stamped -> int_32stamped rather than int_32_stamped.
Fix:
To fix this we can add the condition that we check if the current character is not lower and
not a digit, that we check if the previous character was a lower or digit. If it was a lower
or digit then we add the break.
* fix: Array support for structures
Problem:
The python generated code for handling non-struct and struct vectors
and arrays was inconsistent. The calls to populate the obj api was
creating incorrect code.
Solution:
To fix this the VectorOfStruct and VectorOfNonStruct was rewritten
to handle array cases and bring the two methods in line which each
other.
Testing:
PythonTesting.sh now correctly runs and generates the code for
array_test.fbs.
Minor modifications were done on the test to use the new index
accessor for struct arrays and the script correctly sources the
location of the python code.
* chore: clang format changes
* Added code generated by scripts/generate_code. Modified GetArrayOfNonStruct slightly
to allow for function overloading allowing the user to get a single element of an array
or the whole array.
* Added new_line parameter to OffsetPrefix to allow optional new lines to be added.
This allows us to use the GenIndents method that automatically adds new lines instead.
* Reupload of generated code from the scripts/generate_code.py
* Removed new line in GetVectorAsNumpy.
* Updated Array lengths to use Length methods where possible. Added fallthrough for GenTypePointer. Added digit check to CamelToSnake method. Added and modified tests for ToSnakeCase and CamelToSnake.
* Added range check on the getter methods for vector and array types. Renamed == as is for python
* [C++] Add a failing unit test for #7516 (Rare bad buffer content alignment if sizeof(T) != alignof(T))
* [C++] Fix final buffer alignment when using an array of structs
* A struct can have an arbitrary size and therefore sizeof(struct) == alignof(struct)
does not hold anymore as for value primitives.
* This patch fixes this by introducing alignment parameters to various
CreateVector*/StartVector calls.
* Closes#7516
* Start of mvn-ification of the test
* move to right locations
* Update the IO done in the test to read from resources / write to temp folders
* Add github workflow attempt to mvn test it instead of JavaTest.sh
* Pin the Kotlin benchmark's symlink for /java to the right location
* Inline equality assertions and format JavaTest.java
* fix android gradle source directory
Co-authored-by: Derek Bailey <derekbailey@google.com>
Untyped fixed vectors are not supported in FlexBuffers. There
was an assert to check for it, but on java, asserts are optional. This
change converts the assertion into a runtime exception.
Fixes#7358
Co-authored-by: Derek Bailey <derekbailey@google.com>
* C++: Add option to skip verifying nested flatbuffers
Additionally, add an options struct to the verifier for those
who prefer designated initializers to default arguments. The former
constructor is defined in terms of the latter because in old c++,
having default values for members removes list initialization, making
defining constructors in the other way a lot more challenging to write.
* fixes
* fmt
* formatting, and remove an argument
* fix
Co-authored-by: Casper Neo <cneo@google.com>
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Add timing command to cmakelist
* Start to split test.cpp. Move flexbuffers tests
* Move monster related tests to own file
* Move parser related tests to own file
* Move json related tests to own file
* Move proto related tests to own file
* moved more functions to parser test
* moved more functions to parser test2
* move monster extra tests to monster test
* move evolution tests to own file
* move reflection tests to own file
* move util tests to own file
* rehomed various tests
* move optional scalars test to own file:
* rehome more tests
* move fuzz tests. Got rid of global test_data_path
* fixes for CI failures
* add bazel files
This fixes two problems:
1. The project could be build when using own tags which does not follow the pattern v<major>.<minor>.<patch>.
2. The case "tag points to the commit" will be handled correctly by setting VERSION_COMMIT to 0.
This commit resolves https://github.com/google/flatbuffers/issues/7472
Co-authored-by: Axel Sommerfeldt <axel.sommerfeldt@gmail.com>
Added (for compiler versions that support it):
-Wmissing-declarations
-Wzero-as-null-pointer-constant
Then, fixes to problems identified by the extra warnings
Tested only on GCC 9.4.0
Adjusted the CPP code generator to output nullptr where appropriate,
to satisfy -Wzero-as-null-pointer-constant
Added a lot of 'static' declarations in front of functions,
to satisfy -Wmissing-declarations,
and wrap static function defs in anonymous namespaces.
There are advantages to both anonymous namespaces and static,
it seems that marking a function as static will not publish the name in
the symbol table at all, thus giving the linker less work to do.
With a change introduce in 385dda5c3785ed8d6a35868bc169f07e40e889087fd2edc66,
flatc was not able to emit code for Kotlin if a namespace is specified
and the folders do not exist. The fix create folders if neded.
Additional changes are introduced in gradle files to bring more visibility
to the error messages.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* [C++] Rename template parameter U in make_span of stl_emulation.h
CPPRESTSDK defines a U macro therefore, calls to U() are problematic.
Rename U to W to avoid conflict.
* [C++] Make typenames of span_convertable and make_span more expressive
Co-authored-by: Derek Bailey <derekbailey@google.com>
* Vector of Tables equality
* support nullptr and fix for not being able to use auto in lambda
* use different std::equal overload
* use flatbuffers::unique_ptr
* go back to auto and clang-format fix
Change config.escape_keywords to AfterConvertingCase.
It avoids unecessay escaping since the generated native
structs have fields starting with a uppercase letter
and Go's keywords start with lowercase letters.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* [golang] Add support for text parsing with json struct tags
Add struct tags to Go native structs generated when object API is used.
This allows to use the same JSON file as for C++ text
parsing(i.e. snake_case vs CamelCase) and thus enabling text parsing
for Go(when using object API).
* [golang] Add test for text parsing
Added small test to check and demonstrate text parsing in Go.
Also, ran clang-format on cpp file changes.
* grpc/compiler: Respect filename suffix and extension during code generation
grpc compiler is not respecting filename suffix and extension passed to
flatc CLI. This causes compiler to spit out incorrect code, which then
cannot be compiled without modification.
Following patch fixes the problem.
Note, I ended up removing some code introduced #6954 ("Have grpc include
file with correct filename-suffix given to flatc") in favour of keeping
sanity of the generator code.
Signed-off-by: Aman Priyadarshi <aman.eureka@gmail.com>
* tests: Add filename-suffix and filename-ext test files
* Test 1: Filename extension changed to "hpp".
* Test 2: Filename suffix changed to "_suffix".
* Test 3: Filename extension changed to "hpp" and suffix changed to "_suffix"
Signed-off-by: Aman Priyadarshi <aman.eureka@gmail.com>
* CMake find_package fixes (#7323)
Rename FlatbuffersConfigVersion.cmake to match CMake project name
* CMake find_package fixes (#7323)
Rename FlatBuffersTargets to match CMake project name
* grpc/compiler: Respect filename suffix and extension during code generation
grpc compiler is not respecting filename suffix and extension passed to
flatc CLI. This causes compiler to spit out incorrect code, which then
cannot be compiled without modification.
Following patch fixes the problem.
Note, I ended up removing some code introduced #6954 ("Have grpc include
file with correct filename-suffix given to flatc") in favour of keeping
sanity of the generator code.
Signed-off-by: Aman Priyadarshi <aman.eureka@gmail.com>
* tests: Add filename-suffix and filename-ext test files
* Test 1: Filename extension changed to "hpp".
* Test 2: Filename suffix changed to "_suffix".
* Test 3: Filename extension changed to "hpp" and suffix changed to "_suffix"
Signed-off-by: Aman Priyadarshi <aman.eureka@gmail.com>
Introduce a MSVC_LIKE variable in the CMake scripts, set that variable to
true only if the compiler is either MSVC or tries to emulate the MSVC
command line, and test that variable when setting compiler arguments.
Tested with cmake .. -G Ninja -DCMAKE_C_COMPILER:PATH="clang-cl.exe" -DCMAKE_CXX_COMPILER:PATH="clang-cl.exe" -DFLATBUFFERS_BUILD_TESTS=OFF -DFLATBUFFERS_BUILD_CPP17=ON
Co-authored-by: Kaiyi Li <kaiyili@google.com>
* Move reflection_ts_fbs into a separate directory (#7342)
Right now, reflection_ts_fbs target is in reflection/BUILD.bazel.
This is not ideal because reflection:reflection_fbs_schema is referenced
from :flatc in the root. Thus, for any Bazel projects that want to
include flatbuffers, they need to include npm / yarn_install and nodejs
support all because reflection/BUILD.bazel loads typescript.bzl and that
requires all TypeScript things.
This PR separated that target into a different subdirectory, freeing
root BUILD.bazel from that dependency.
* Minor improvements to typescript.bzl
* Uses @...// dependencies so that flatbuffers can actually
be used as an external repo (had forgotten to upstream this earlier).
* Allows using flatbuffer_ts_library to generate reflection schemas
in the same way that flatbuffer_cc_library does (but default it
to off to avoid behavioral changes).
* Pass through a package_name attribute to flatbuffer_ts_library to
allow non-relative imports of generated typescript code.
Co-authored-by: Liu Liu <i@liuliu.me>
Presently flatc generates a placeholder for the unused synchronous varient of client-side streaming gRPCs which includes the name of an unused parameter in the method.
To avoid warnings, comment out the parameter name (in a manner similar to other gRPCS).
The logic to manage generating typescript in a single file was
generating self imports and unused local names, which triggered some
linters.
This resolves#7191
There was a step in the compilation process where benchmark data is
downloaded before starting the kotlin compilation process.
Since we are not running benchmark on CI anymore, we remove the
dependency. To run benchmarks the download task needs to be executed
manually.
* Implement optional scalars for JSON
* Add optional scalars JSON test
* Extend JSON optional scalars test to test without defaults
* Fix optional scalars in JSON for binary schema
Co-authored-by: Caleb Zulawski <caleb.zulawski@caci.com>
Right now, reflection_ts_fbs target is in reflection/BUILD.bazel.
This is not ideal because reflection:reflection_fbs_schema is referenced
from :flatc in the root. Thus, for any Bazel projects that want to
include flatbuffers, they need to include npm / yarn_install and nodejs
support all because reflection/BUILD.bazel loads typescript.bzl and that
requires all TypeScript things.
This PR separated that target into a different subdirectory, freeing
root BUILD.bazel from that dependency.
* Implement optional scalars for Python
* Use == for integer comparison, remove empty line
* Fix optional type hint
Co-authored-by: Caleb Zulawski <caleb.zulawski@caci.com>
Fixes#7320.
I realize that Rust doesn't really follow the Namer convention, since
it uses Field case for methods... that's a future problem.
Co-authored-by: Casper Neo <cneo@google.com>
* Add explicit return types to lobster generated code
* Add support for optional fields.
Convert to bool explicitly from int8 to match type signature
Fix whitespace
I think these changes reflect the current state, but I found it hard to
track the state of some of the planned work. Hence why it'd be good to
have it documented :)
I also expanded some sections that I found misleading, as somebody
familiar with Rust and FlatBuffers separately. Parts of these pages seem
to be aimed at people familiar with FlatBuffers (ie via the other
documentation pages) but not each language, which I'm trying to
preserve. However, Rust does some things differently, and as somebody
with expectations about how typical Rust APIs work the discussion of
threading made me wonder what was different.
std::span lacks these; make the flatbuffers STL emulation and tests
match. This fixes a compile error in C++20 mode when using std::span.
Bug: chromium:1284275
* Set an explicit 2018 edition for Rust tests
* Replace all `std` usage with `core` and `alloc` in Rust code generator
* Update the generated files
* Make Rust tests actually use no_std when the corresponding feature is enabled
* Keep the underlying storage capacity when clearing the FlatBufferBuilder. Gives a significant performance boost for serialisation of many small messages.
* Use Googles Swift benchmark library for more consistent results and dynamic number of iterations, simplification of tests as result.
Co-authored-by: Joakim Hassila <hassila@users.noreply.github.com>
Since CreateVectorOfStrings() takes a templated container, make sure that
the default template deduction from just an initializer list will
still work.
Signed-off-by: Henner Zeller <hzeller@google.com>
* [Java] Fix key lookup returning null clashing with default value
A field with key attribute must always be written on the message so it
can be looked up by key. There is a edge case where inserting a key
field with same value as default would prevent it to be written on
the message and later cannot be found when searched by key.
* [Kotlin] Fix key lookup returning null clashing with default value
A field with key attribute must always be written on the message so it
can be looked up by key. There is a edge case where inserting a key
field with same value as default would prevent it to be written on
the message and later cannot be found when searched by key.
Co-authored-by: Derek Bailey <derekbailey@google.com>
A field with key attribute must always be written on the message so it
can be looked up by key. There is a edge case where inserting a key
field with same value as default would prevent it to be written on
the message and later cannot be found when searched by key.
Any string type that is supported by CreateString(), e.g.
const char* or string_view will now also work.
Signed-off-by: Henner Zeller <hzeller@google.com>
* [Kotlin] Update gradle to 7.4.1 and simplify config files.
* [Kotlin] Add wrapper-validation-action to Kotlin Linux
* [Kotlin] Remove benchmark actions to reduce CI time
* [Kotlin] Move CI js test to Linux action, to increase Mac action speed
* [Kotlin] Generate gradle wrapper in order to be validate
Gradle wrapper from 3.3 to 4.0 are not verifiable because those files
were dynamically generated by Gradle in a non-reproducible way.
So they are now regenerated and will be checked using gitlab action:
gradle/wrapper-validation-action@v1
* Adds implementation flag for swift
Forces internal flag when using @_implementationOnly in swift
Fixes access type for verifier functions & encoder functions
Updates generated code
* Addresses PR comments & adds a code gen dir within the swift tests
* Adds test case for no-include
* Fixes code gen script
Removes prefix
* Started to migrate to target_compile_options
* combined compile options together. Added Mac CI builds
* remove arm build (not supported). Fixed old-style-casts
* moved to using a ProjectConfig interface library to specify options
* remove the explicit CMAKE_CXX_STANDARD
* code gen flexbuffer verifier
* remove verify nested flexbuffers from flexbuffers
* made function static, and placed higher in file
* moved function to own header
* Typo in flatc options (warning-as-errors instead of warnings-as-errors)
* VerifySizePrefixed (reflection::Schema) and GetAnySizePrefixedRoot added
* some review comments
* more review comments
* Fix for https://github.com/google/flatbuffers/issues/7209
* Fixes [C++] flatc generates invalid Code for union field accessors, when --cpp-field-case-style is used #7210
* Add overloads to Add/Put for ArraySegment and IntPtr
In order to allow using code to reduce memory allocations, add overloads to ByteBuffer's and FlatBuffersBuilder's Put/Add methods that take ArraySegment<T> or IntPtr respectively.
Also, adaptions to the c# code generator in flatc to emit corresponding CreateVectorBlock() overloads
* Add missing files generated with generate_code.py
The previous commit changed the C# code generate, but didn't contain the updated generated test files.
* Incorporate review findings
(1) Adhere to 80 characters limit.
(2) In FlatBufferBuilder.Add(IntPtr,int), move zero length check topmost and add sanity check against negative input
* Started applying Namer to Java.
- Java didn't previously have keyword escaping
- Added prefixes and suffixes to the Namer methods
- TODO: migrate previous namer applications to using pre/suffixes
- Java methods / functions are interesting, it's mostly camel case
except when it involves a struct/enum name. That section is Keep case
- I changed the casing for some internal arguments/variables. This
violates the "don't change genfiles" rule that I've been using but it
shouldn't break user code.
- LegacyJavaMethod2 is interesting. Basically, Java has a "mixed" case
convention where it's camel case, except for the type/variant name
itself, which is keep case. So a type foo_bar would become getfoo_bar
instead of getFooBar.
* small fix
* Namer for Namespaces
* removed unused parameter, add const everywhere
* Remove unused argument
* More unused args
* Use mutable reference out parameters
* Made more strings const and inlined const empty strings
* remove do not submit
Co-authored-by: Casper Neo <cneo@google.com>
* Fix C/C++ Create<Type>Direct with sorted vectors
If a struct has a key the vector has to be sorted. To sort the vector
you can't use "const".
* Changes due to code review
* Improve code readability
* Add generate of JSON schema to string to lib
* option indent_step is supported
* Remove unused variables
* Fix break in test
* Fix style to be consistent with rest of the code
* [TS] Fix reserved words as arguments (#6955)
* [TS] Fix generation of reserved words in object api (#7106)
* [TS] Fix generation of object api
* [TS] Fix MakeCamel -> ConvertCase
* [C#] Fix collision of field name and type name
* [TS] Add test for struct of struct of struct
* Update generated files
* Add missing files
* [TS] Fix query of null/undefined fields in object api
* Add example for type field name collision
* [swift] Add bazel configuration for Swift
This change adds a simple bazel BUILD file for consuming the Swift
support. This also bumps the platforms bazel repo to fix support for M1s
and bazel 5.1+ https://github.com/bazelbuild/bazel/issues/15099#issuecomment-1075368289
The rules_swift inclusion here must happen before gRPC to ensure we
don't pull in an older version.
* Add CC=clang which is a requirement for Swift on Linux
* Add Swift to PATH
* Apply Namer to Dart.
- Also refactor idl_gen_dart a bit
- to use more const and references
- out parameters should be the last argument
* Add keyword test
* minor fixes
* fix merge
* extra 's'
* move dart keyord into dart dir
* Address comments
* Use $ for escaping keywords
* Outparameters for namespace_map
* Escape dollar in toString
* Escape dollar in toString2
* Use UpperCamelCase for types and variants
* try to fix ToString
* namer Type fixes
* Remove path prefixing in imports
* gen code
Co-authored-by: Casper Neo <cneo@google.com>
* Handle invalid root offset
* Handle vtable offset invalidation
* Added script generator. Add more cases through vtable ref table size
* review responses
* vtable offset validation
* Moved padding insertion to the end. Tests invalid field lenghts
* table offsets validated. Added type after field
* validate string length
* add todo
* invalid vector length
* invalid structs
* general cleanup
* reworded invalid offsets
* example for vector of structs
* invalid vector of tables
* invalid vector of strings
* invalid vector of scalars
* vector of unions
* validate union type value
* invalid vector union type values
* Fix C/C++ Create<Type>Direct with sorted vectors
If a struct has a key the vector has to be sorted. To sort the vector
you can't use "const".
* Changes due to code review
* Improve code readability
* Add generate of JSON schema to string to lib
* option indent_step is supported
* Remove unused variables
* Fix break in test
* Fix style to be consistent with rest of the code
* [TS] Fix reserved words as arguments (#6955)
* [TS] Fix generation of reserved words in object api (#7106)
* [TS] Fix generation of object api
* [TS] Fix MakeCamel -> ConvertCase
* [TS] Add test for struct of struct of struct
* Update generated files
* Add missing files
* [TS] Fix query of null/undefined fields in object api
* Typo in flatc options (warning-as-errors instead of warnings-as-errors)
* VerifySizePrefixed (reflection::Schema) and GetAnySizePrefixedRoot added
* some review comments
* more review comments
* Apply Namer to Lua bfbs code gen
* refactor namer into IdlNamer to keep idl includes separate
* remove commented out code
* added bfbs_namer
* remove Enum case
* add to bazel
* Allow suppression of including <optional> via macro
* Combine preprocessor conditions
* Make inclusion of header <optional> opt-in
* Make inclusion of <optional> opt-out via macro
* Auto-detect C++17+ when defining macro
* Enclose macro expression in parentheses
* Check value of macro, not whether it's defined
* Fix parentheses
* Don't define macro with expression using other macros
The preprocessor doesn't seem to like it.
* Fix parentheses
* Define and use Namer overloads
* more
* NamespacedType needs a public string-like overload
* Move Rust FieldOffsetName to Namer LegacyRustFieldOffsetName
* Started swift
* More Namer updates
* Remove more usage of the variable 'name' which was semantically overloaded
* unshadow varible
* Make ptr to bool explicit
Co-authored-by: Casper Neo <cneo@google.com>
* Define and use Namer overloads
* more
* NamespacedType needs a public string-like overload
* Move Rust FieldOffsetName to Namer LegacyRustFieldOffsetName
Co-authored-by: Casper Neo <cneo@google.com>
The headline here is adding a flatbuffer_ts_library rule for generating
typescript code in bazel. This entails some non-trivial other changes,
but ideally none are user-visible.
In particular:
* Added a --ts-flat-file flag that generates a single *_generated.ts
file instead of separate files for each typescript type. This makes
bazel much happier.
* Import the bazel rules_nodejs stuff needed to support building
typescript in bazel
* Move flatbuffers.ts to index.ts because I wasn't sure how to make
bazel comprehend the "main" attribute of the package.json. Happy
to take another stab at figuring that out if really needed.
* Fix another couple keyword escaping spots in typescript...
* Refactor out a class from Rust Codegen
* Convert GenerateRustModuleRootFile
* git-clang-format
* unused variable
* parenthesis
* update BUILD file
* buildifier
* Delete bfbs_gen_rust.h
* Delete bfbs_gen_rust.cpp
* Addressed some comments
* Namer::EnumVariant
* Remove do not submit; Add Namespace vector overload
* Unshadow variable
* removed redundant variables
* Apply Namer to Python
* Use more variables a bit
* Apply const a bunch
* More variables
* Fix ObjectTypes
* git clang format
* small thing
* Simplified code around nested flatbuffers
* Make more methods const.
* Python files are kKeep case
* Address DO NOT SUBMIT in SaveType
* ensure dir exists before saving files
* clang format
Co-authored-by: Casper Neo <cneo@google.com>
* Refactor out a class from Rust Codegen
* Convert GenerateRustModuleRootFile
* git-clang-format
* unused variable
* parenthesis
* update BUILD file
* buildifier
* Delete bfbs_gen_rust.h
* Delete bfbs_gen_rust.cpp
* Addressed some comments
* Namer::EnumVariant
* Remove do not submit; Add Namespace vector overload
* Unshadow variable
* removed redundant variables
* Apply Namer to Python
* Use more variables a bit
* Apply const a bunch
* More variables
* Fix ObjectTypes
* git clang format
* small thing
* Simplified code around nested flatbuffers
* Make more methods const.
* Python files are kKeep case
* Address DO NOT SUBMIT in SaveType
* ensure dir exists before saving files
* fix space
Co-authored-by: Casper Neo <cneo@google.com>
* Add Object to the list of reserved keywords.
* Properly escape keywords in type names.
* Properly escape keywords in enum names.
* Properly escape keywords in enum field names.
* Fix 64-bit default numeric enum values in typescript
If you had a default value that wasn't a valid enum value (e.g., a zero
if you used a bit_flag setting, like you get with AdvancedFeatures
in reflection.fbs), we weren't using BigInt.
* Run generate_code.py
* [DART] Handle deprecated fields & invalid enum defaults
* Update .NET test
* [Rust] Add length checks to arrays and vectors.
The previous behavior allowed for out of bounds access in
the public API (#7011).
* bump semver and test warning
Co-authored-by: Casper Neo <cneo@google.com>
* Unified name case conversion to single method
* Convert bfbs_gen to use ConvertCase
* convert rust to use ConvertCase
* Convert idl_parser to use ConvertCase
* Convert MakeScreamingCamel to ConvertCase
* Replaced MakeCamel with ConvertCase
* minor fixes
I'm not seeing the reason why we didn't attempt to support transitive
dependencies for flatbuffer_cc_library, and the current setup makes
having to propagate new dependencies to all of their recursive
dependents obnoxious.
* Fix C/C++ Create<Type>Direct with sorted vectors
If a struct has a key the vector has to be sorted. To sort the vector
you can't use "const".
* Changes due to code review
* Improve code readability
* Add generate of JSON schema to string to lib
* option indent_step is supported
* Remove unused variables
* Fix break in test
* Fix style to be consistent with rest of the code
* [TS] Fix reserved words as arguments (#6955)
* [TS] Fix generation of reserved words in object api (#7106)
* [go] always write required types
* support optional scalars in go
* generate optional_scalars and monster_test
* restore original behavior for non-optional scalars
* add tests
* Change Rust generated file defaults
After #6731, flatc changed its default behavior
for generating rust code to fix some importing issues.
This was a breaking change which invlidated the patch release,
`flatc 2.0.5` (#7081). This PR reverses the default so we can
release a patch update. However, does break Rust users who work at
HEAD.
* Bump flatc patch version (2.0.6)
Co-authored-by: Casper Neo <cneo@google.com>
We already have the reflection.fbs file and the flatbuffers
python language support.
Adding this feature would give the python developers the
ability to parse the flatbuffers schema and write some tools.
Co-authored-by: Derek Bailey <derekbailey@google.com>
* fix for rust build
* Rust: Implement Serialize on generated types
For debugging convenience it is really handy to be able to dump out
types as another format (ie: json). For example, if we are logging a
type to a structured logging system, or even printing it out in a
structured way to the console.
Right now we handle this by shelling out to `flatc` which is not ideal;
by implementing Serialize on the generated types we can use any of the
Serializer-implementing packages for our structured debug output.
* clang-format
* Make the flatbuffers Rust crate only have an optional dependency on the `serde` packages.
* fix warning
* fix rust test build
* Oh yeah this needs to be initialized
* fix toml syntax
* code review feedback
* rebuild test data
Augment the C++ generator to emit a C++ copy constructor and a by-value
copy assignment operator. This is enabled by default when the C++
standard is C++11 or later. These additional functions are only emitted
for objects that need it, typically tables containing other tables.
These new functions are declared in the object table type and are
defined as inline functions after table declarations.
When these new functions are declared, a user-defined
explicitly-defaulted default constructor and move constructor are also
emitted.
The copy assignment operator uses the copy-and-swap idiom to provide
strong exception safety, at the expense of keeping 2 full table copies
in memory temporarily.
fixes#5783
* [ts] Builder incorrectly serializing empty strings
The builder was returning an offset of zero for empty strings. This is
leading to flatbuffers which fail verification in other languages, such
as Rust.
* tests expect 0 offset for null or undefined strings
* Emit include for bfbs-gen-embed
* Use python3 explicitly
* bump min python version to 3.6
* Sort find_package for python
* try casting Path to string
* cast WindowsPath to string to please CI
* stringify the wrong thing
* another stringify path
* Moved the CMake check for Clang *after* the one for MSVC, as clang-cl
matches both Clang and MSVC.
* Removed /MP definition - controlled by CMake
* Added CPP define for _CRT_SECURE_NO_WARNINGS so clang can see it
(it seems it doesnt pick up the pragma in util.cpp)
* BuildFlatbuffers.cmake: add verbose on build
* BuildFlatbuffers.cmake: properly add *.fb.* files with --grpc argument
When "--grpc" argument is provided as an extra flag, resulting grpc files
should be added as part of the interface library.
This prevent adding .fb.cc files manually to the build.
V2: fix dependency on grpc files
* Add --warnings-as-errors to flatc compiler.
With this option set, flatc will return an error on parsing if
any warnings occurred.
* Add unit test for opts.warnings_as_errors.
* Change explicit option setting to default.
https://github.com/tensorflow/tflite-micro makes use of flatbuffers with
a variety of DSP toolchains.
Without the change from this PR, we can get a double-promotion warning
with some of these DSP toolchains:
```
flatbuffers/include/flatbuffers/util.h:104:11: error: implicit conversion increases floating-point precision: 'std::numeric_limits<float>::_Ty' (aka 'float') to 'double' [-Werror,-Wdouble-promotion]
T eps = std::numeric_limits<float>::epsilon();
~~~ ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```
iomanip isn't available for our DSP. Luckily, we compile with FLATBUFFERS_PREFER_PRINTF, so moved the #include directive there.
ctype.h has to be included explicilty for tolower() and toupper().
Parsing as bytes produces buffers that are unsafe to access unless passed thru a verifier,
whereas users could reasonably assume that any JSON parsed without errors is safe to access.
Users that still have legacy JSON files with such bytes in it will get a helpful error point them
to the option to turn on to have it work again.
This change checks if the current source directory is a git repository.
If this is not checked the git command picks the commit hash from
parent directory. e.g. when tarball is extracted in a packaging repository.
* Enable --gen-onefile in Python
Made it possible to generate all python code in one file. Modified
py_test.py so that it can switch between the multi-file code and
the one-file code.
Updated PythonTest.sh and py_test.py so that the multi-file code
and the one-file code can be tested based on the same test code.
* Sync with google/flatbuffers
* Add --gen-onefile to generate_code.py
* Make idl_parser deterministic
Some golden tests even exercise this [logic](df2df21ec1/tests/prototest/test.golden (L8)). Let's make the parser fully stable not depending on the implementation of std::sort
* Retry the ci
2021-12-09 13:07:29 -08:00
1813 changed files with 210457 additions and 68521 deletions
Please delete this standard text once you've created your own description.
If you make changes to any of the code generators (`src/idl_gen*`) be sure to
build your project, as it will generate code based on the changes. If necessary
[build](https://google.github.io/flatbuffers/flatbuffers_guide_building.html) your project, as it will generate code based on the changes. If necessary
the code generation script can be directly run (`scripts/generate_code.py`),
requires Python3. This allows us to better see the effect of the PR.
stale-pr-message:'This pull request is stale because it has been open 6 months with no activity. Please comment or this will be closed in 14 days.'
stale-issue-message:'This issue is stale because it has been open 6 months with no activity. Please comment or this will be closed in 14 days.'
days-before-stale:182# 6 months
days-before-close:14
operations-per-run:1500
exempt-issue-labels:not-stale
exempt-pr-labels:not-stale
operations-per-run:500
exempt-all-milestones:true
remove-stale-when-updated:true
stale-issue-message:'This issue is stale because it has been open 6 months with no activity. Please comment or label `not-stale`, or this will be closed in 14 days.'
close-issue-message:'This issue was automatically closed due to no activity for 6 months plus the 14 day notice period.'
days-before-issue-stale:182# 6 months
days-before-issue-close:14# 2 weeks
exempt-issue-labels:not-stale
stale-pr-message:'This pull request is stale because it has been open 6 months with no activity. Please comment or label `not-stale`, or this will be closed in 14 days.'
close-pr-message:'This pull request was automatically closed due to no activity for 6 months plus the 14 day notice period.'
* flatc: `--grpc-callback-api` flag generates C++ gRPC Callback API server `CallbackService` skeletons AND client native callback/async stubs (unary + all streaming reactor forms) (opt-in, non-breaking, issue #8596).
* Swift - Adds new API to reduce memory copying within swift (#8484)
* Rust - Support Rust edition 2024 (#8638)
* [C++] - Use the Google Style for clang-format without exceptions (#8706)
* Migrate from rules_nodejs to rules_js/rules_ts (take 2) (#7928)
*`flat_buffers.dart`: mark const variable finals for internal Dart linters
* fixed some windows warnings (#7929)
* inject no long for FBS generation to remove logs in flattests (#7926)
* Revert "Migrate from rules_nodejs to rules_js/rules_ts (#7923)" (#7927)
* Migrate from rules_nodejs to rules_js/rules_ts (#7923)
* Only generate @kotlin.ExperimentalUnsigned annotation on create*Vector methods having an unsigned array type parameter. (#7881)
* additional check for absl::string_view availability (#7897)
* Optionally generate Python type annotations (#7858)
* Replace deprecated command with environment file (#7921)
* drop glibc from runtime dependencies (#7906)
* Make JSON supporting advanced union features (#7869)
* Allow to use functions from `BuildFlatBuffers.cmake` from a flatbuffers installation installed with CMake. (#7912)
* TS/JS: Use TypeError instead of Error when appropriate (#7910)
* Go: make generated code more compliant to "go fmt" (#7907)
* Support file_identifier in Go (#7904)
* Optionally generate type prefixes and suffixes for python code (#7857)
* Go: add test for FinishWithFileIdentifier (#7905)
* Fix go_sample.sh (#7903)
* [TS/JS] Upgrade dependencies (#7889)
* Add a FileWriter interface (#7821)
* TS/JS: Use minvalue from enum if not found (#7888)
* [CS] Verifier (#7850)
* README.md: PyPI case typo (#7880)
* Update go documentation link to point to root module (#7879)
* use Bool for flatbuffers bool instead of Byte (#7876)
* fix using null string in vector (#7872)
* Add `flatbuffers-64` branch to CI for pushes
* made changes to the rust docs so they would compile. new_with_capacity is deprecated should use with_capacity, get_root_as_monster should be root_as_monster (#7871)
* Adding comment for code clarification (#7856)
* ToCamelCase() when kLowerCamel now converts first char to lower. (#7838)
* Fix help output for --java-checkerframework (#7854)
* Update filename to README.md and improve formatting (#7855)
@@ -19,4 +19,4 @@ Swift uses swiftformat as it's formatter. Take a look at [how to install here](h
## Typescript
Typescript uses eslint as it's linter. Take a look at [how to install here](https://eslint.org/docs/user-guide/getting-started). Run the following command `eslint ts/** --ext .ts` in the root directory of the project
Typescript uses eslint as it's linter. Take a look at [how to install here](https://eslint.org/docs/user-guide/getting-started). Run the following command `eslint ts/** --ext .ts` in the root directory of the project
**FlatBuffers** is a cross platform serialization library architected for
maximum memory efficiency. It allows you to directly access serialized data without parsing/unpacking it first, while still having great forwards/backwards compatibility.
## Quick Start
1. Build the compiler for flatbuffers (`flatc`)
Use `cmake` to create the build files for your platform and then perform the compilation (Linux example).
```
cmake -G "Unix Makefiles"
make -j
```
2. Define your flatbuffer schema (`.fbs`)
Write the [schema](https://flatbuffers.dev/flatbuffers_guide_writing_schema.html) to define the data you want to serialize. See [monster.fbs](https://github.com/google/flatbuffers/blob/master/samples/monster.fbs) for an example.
3. Generate code for your language(s)
Use the `flatc` compiler to take your schema and generate language-specific code:
```
./flatc --cpp --rust monster.fbs
```
Which generates `monster_generated.h` and `monster_generated.rs` files.
4. Serialize data
Use the generated code, as well as the `FlatBufferBuilder` to construct your serialized buffer. ([`C++` example](https://github.com/google/flatbuffers/blob/master/samples/sample_binary.cpp#L24-L56))
5. Transmit/store/save Buffer
Use your serialized buffer however you want. Send it to someone, save it for later, etc...
6. Read the data
Use the generated accessors to read the data from the serialized buffer.
It doesn't need to be the same language/schema version, FlatBuffers ensures the data is readable across languages and schema versions. See the [`Rust` example](https://github.com/google/flatbuffers/blob/master/samples/sample_binary.rs#L92-L106) reading the data written by `C++`.
## Documentation
**Go to our [landing page][] to browse our documentation.**
## Supported operating systems
- Windows
- macOS
- Linux
- Android
- And any others with a recent C++ compiler (C++ 11 and newer)
## Supported programming languages
Code generation and runtime libraries for many popular languages.
1. C
1. C++ - [snapcraft.io](https://snapcraft.io/flatbuffers)
FlatBuffers does not follow traditional SemVer versioning (see [rationale](https://github.com/google/flatbuffers/wiki/Versioning)) but rather uses a format of the date of the release.
## Contribution
* [FlatBuffers Issues Tracker][] to submit an issue.
* [stackoverflow.com][] with [`flatbuffers` tag][] for any questions regarding FlatBuffers.
*To contribute to this project,* see [CONTRIBUTING][].
## Community
* [Discord Server](https:///discord.gg/6qgKs3R)
## Security
Please see our [Security Policy](SECURITY.md) for reporting vulnerabilities.
## Licensing
*Flatbuffers* is licensed under the Apache License, Version 2.0. See [LICENSE][] for the full license text.
# Workaround for https://github.com/conda/conda-build/issues/636
PYTHONIOENCODING:UTF-8
CONDA_INSTALL_LOCN:"C:\\Miniconda35-x64"
CMAKE_OPTIONS:""
CPP_TEST_OPTIONS:""
CHECK_GEN_CODE:1
matrix:
- APPVEYOR_BUILD_WORKER_IMAGE:Visual Studio 2015
CMAKE_VS_VERSION:"14 2015"
platform:
- x86
- x64
configuration:
- Debug
- Release
before_build:
- cmake . -G"Visual Studio %CMAKE_VS_VERSION%" -DFLATBUFFERS_CODE_SANITIZE=1 %CMAKE_OPTIONS%
# This cuts down on a lot of noise generated by xamarin warnings.
- if exist "C:\Program Files (x86)\MSBuild\14.0\Microsoft.Common.targets\ImportAfter\Xamarin.Common.targets" del "C:\Program Files (x86)\MSBuild\14.0\Microsoft.Common.targets\ImportAfter\Xamarin.Common.targets"
build:
project:ALL_BUILD.vcxproj
verbosity:minimal
after_build:
- python conan/appveyor/install.py
- python conan/appveyor/build.py
install:
- set PATH=%CONDA_INSTALL_LOCN%;%CONDA_INSTALL_LOCN%\scripts;%PATH%;
- ps:Install-Product node $env:nodejs_version
test_script:
- if [%CHECK_GEN_CODE%]==[1] (call py scripts/check_generate_code.py "%CPP_TEST_OPTIONS%")
- "cd tests"
- 7z a GeneratedMyGameCode.zip MyGame\
- rem "---------------- C++ -----------------"
- "cd .."
- "%CONFIGURATION%\\flattests.exe"
- rem "---------------- JS -----------------"
- "node --version"
- "npm install"
- "npm run compile"
- "cd tests"
- "TypeScriptTest.bat"
- rem "---------------- C# -----------------"
# Have to compile this here rather than in "build" above because AppVeyor only
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
