There are a couple instances where the ByteBuffer's Span property was accessed in a loop.
+ Extracted the use of the property outside of the loop to save a few cpu cycles.
Access to the allocator's internal buffer isn't exposed as a ReadOnlySpan<byte> from the ByteBuffer
or the FlatBufferBuilder.
+ Added a few convenience functions to access the buffer using a ReadOnlySpan<byte>.
There are a few cases where built in Span extensions can be used to run optimized code.
+ Added the use of Span.Fill() and ReadOnlySpan.SequenceCompareTo to replace existing loops.
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
new_p is a local addr but is owned now by slice_
thus the life time does not end at the end of the function
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
* fix(go/grpc): avoid panic on short FlatBuffers input
The gRPC codec read the root UOffsetT without checking input size. On
buffers shorter than SizeUOffsetT, GetUint32 touched data[3] and the
process panics.
Add a simple length check and validate the root offset stays within the
buffer. Return clear errors (insufficient data / invalid root offset)
instead of panicking.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
* fix(go/grpc): avoid signed overflow in offset
Keep the bounds check in the unsigned domain (UOffsetT) to avoid
signedness pitfalls.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
* chore: clarify comment regarding offset
A full FlatBuffer structure would be:
- uoffset_t: root table offset (4 bytes)
- soffset_t: vtable offset in root table (4 bytes)
- uint16_t: vtable size (2 bytes)
- uint16_t: table size (2 bytes)
In total 12 bytes. We are only validating the data length
before trying to read the uoffset_t, not the full structure.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
---------
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
Co-authored-by: Derek Bailey <derekbailey@google.com>
Our CI is broken and this is the error:
```
FAILURE: Build failed with an exception.
* What went wrong:
Gradle requires JVM 17 or later to run. Your build is currently configured to use JVM 11.
```
So updating our java-versions to the latest stable version which is 21 apparently.
Makes the return type of `static getFullyQualifiedName()` be a string literal instead of just the string type
Update tests
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
* [Python] Sync PythonTest.sh flags with generate_code.py
* [Python] Update generated code to latest flatc version for tests
* [Python] Fix test support for numpy newer than 2.0.0
* [Python] Remove unused variable
* [Python] Fix __eq__ for numpy arrays
* [Python] Run clang-format over the entire file
* Developers intro how to contribute
* Fix Rust code generation for Rust edition 2024
The errors look like:
```
warning[E0133]: call to unsafe function `fbs::flatbuffers::emplace_scalar` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::follow_cast_ref` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::Follow::follow` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::read_scalar_at` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::root_unchecked` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::size_prefixed_root_unchecked` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `fbs::flatbuffers::Table::<'a>::new` is unsafe and requires unsafe block
warning[E0133]: call to unsafe function `std::slice::from_raw_parts` is unsafe and requires unsafe block
```
* Update goldens
Ran `goldens/generate_goldens.py`
* Regenerate code files
Ran `scripts/generate_code.py`
This commit significantly improves the developer experience for the Python Object-Based API by overhauling the generated `__init__` method for `T`-suffixed classes.
Previously, `T` objects had to be instantiated with an empty constructor, and their fields had to be populated manually one by one. This was verbose and not idiomatic Python.
This change modifies the Python code generator (`GenInitialize`) to produce `__init__` methods that are:
1. **Keyword-Argument-Friendly**: The constructor now accepts all table/struct fields as keyword arguments, allowing for concise, single-line object creation.
2. **Fully Typed**: The signature of the `__init__` method is now annotated with Python type hints. This provides immediate benefits for static analysis tools (like Mypy) and IDEs, enabling better autocompletion and type checking.
3. **Correctly Optional**: The generator now correctly wraps types in `Optional[...]` if their default value is `None`. This applies to strings, vectors, and other nullable fields, ensuring strict type safety.
The new approach remains **fully backward-compatible**, as all arguments have default values. Existing code that uses the empty constructor will continue to work without modification.
#### Example of a Generated `__init__`
**Before:**
```python
class KeyValueT(object):
def __init__(self):
self.key = None # type: str
self.value = None # type: str
```
**After:**
```python
class KeyValueT(object):
def __init__(self, key: Optional[str] = None, value: Optional[str] = None):
self.key = key
self.value = value
```
#### Example of User Code
**Before:**
```python
# Old, verbose way
kv = KeyValueT()
kv.key = "instrument"
kv.value = "EUR/USD"
```
**After:**
```python
# New, Pythonic way
kv = KeyValueT(key="instrument", value="EUR/USD")
```
Using the : syntax leads to non member attributes.
> If an attribute is defined in the class body with a type annotation
> but with no assigned value, a type checker should assume this is a non-member attribute
```
class Pet(Enum):
genus: str # Non-member attribute
species: str # Non-member attribute
CAT = 1 # Member attribute
DOG = 2 # Member attribute
```
https://typing.python.org/en/latest/spec/enums.html#defining-members
This prevents the include of the type defined in the pyi,
otherwise this leads to error message like this:
error: Name XYZ already defined (possibly by an import) [no-redef]
* [Python] Use correct type for str with None
Otherwise mypy will correctly flag code like this
def __init__(self):
self.fooBar = None # type: Optional[str]
error: Incompatible types in assignment (expression has type "None", variable has type "str")
* [Python] Make list type optional as they can contain None
Fixes an issue where exports were using incorrect relative paths for
>=3 namespace levels. This is fixed by making the starting range of the
namespace components relative to the amount of components.
Co-authored-by: Björn Harrtell <bjornharrtell@users.noreply.github.com>
Adding support for windows requires the code generations
to add a compiler statement to completely ignore GRPC code
generation on windows
Cleanup the project to use the main Package.swift to run tests
instead of having it separate and includes the imports for GRPC
within it.
Adds windows swift ci
* clang-format
* [Python] Replace . with _ in grpc filename suffix
Having filenames with . like `file.fb.grcp`
is not great for Python. Since dots are used for namespaces.
Replacing all of them with _ eg suffix `foo.bar.baz` will become
`foo_bar_baz`.
Restoring the previous default `_fb` suffix.
* [Python] Use namespace in path
This fixes a regression introduced with:
fb9afbafc7
And generates the grpc file in the namespace folder again.
* Sync commandline docs with web docs
* Offical Swift port for FlexBuffers
This is the offical port for FlexBuffers within
swift, and it introcudes a Common Module where code
is shared between flatbuffers and flexbuffers.
Writing most supported values like maps, vectors,
nil and scalars into a flexbuffer buffer. And includes
tests to verify that its similar to cpp
* Reading a flexbuffer
Implementing reading from a flexbuffer, enabling
most of the buffers features, like most types, maps, vectors,
typedvectors, and fixedtypedvectors.
Currently, if an offset/object cant be read we default to a swift
nil instead of the default flexbuffers 'null' with all values.
* Fixes bazel breaking due to new project structure
Address warnings within the library
* Adds comment on why we added the code & properly enforce the amout of bytes needed
For Vector or Array of structures the dereference operator of an
iterator returns the pointer to the structure. However, IndirectHelper,
which is used in the implementation of this operator, is instantiated
in the way that the IndirectHelper::Read returns structure by value.
This is because, Vector and Array instantiate IndirectHelper with
const T*, but VectorIterator instantiates IndirectHelper with T. There
are three IndirectHelper template definition: first for T, second for
Offset<T> and the last one for const T*. Those have different
IndirectHelper:Read implementations and (more importantly) return type.
This is the reason of mismatch in VectorIterator::operator* between
return type declaration and what was exactly returned.
That is, for Array<T,...> where T is scalar the VectorIterator is
instantiated as VectorIterator<T, T>, dereference operator returns T
and its implementation uses IndirectHelper<T> which Read function
returns T.
When T is not scalar, then VectorIterator is instantiated as
VectorIterator<T, const T *>, dereference operator returns const T * and
its implementation uses IndirectHelper<T> which Read function returns T.
The fix is done as follows:
* implement type trait is_specialization_of_Offset and
is_specialization_of_Offset64,
* change partial specialization of IndirectHelper with const T * that
it is instantiated by T and enabled only if T is not scalar and not
specialization of Offset or Offset64,
* remove type differentiation (due to scalar) from Array..
The above makes the IndirectHelper able to correctly instantiate itself
basing only on T. Thus, the instantiation in VectorIterator correctly
instantiate IndirectHelper::Read function, especially the return type.
Fixes access to union members when generating code with options "--cpp-field-case-style upper" and "--gen-object-api"
Co-authored-by: Wouter van Oortmerssen <aardappel@gmail.com>
* Adds new API to reduce memory copying within swift
Adds new storage container _InternalByteBuffer which
will be holding the data that will be created within the swift
lib, however reading data will be redirected to ByteBuffer, which
should be able to handle all types of data that swift provide without
the need to copy the data itself. This is due to holding a reference to
the data.
Replaces assumingMemoryBinding with bindMemory which is safer
Adds function that provides access to a UnsafeBufferPointer for
scalars and NativeStructs within swift
Updates docs
Suppress compilation warnings by replacing var with let
Using overflow operators within swift to improve performance
Adds tests for GRPC message creation from a retained _InternalByteBuffer
