* Fix generate_code script path
* [Python] Make StartVector public
Make StartVector vector public since it is already being used in
generated code
* [Python] Improve vector creation for Python API
Makes Python API for vectors cleaner like Rust and Swift
---------
Co-authored-by: Derek Bailey <derekbailey@google.com>
This commit significantly improves the developer experience for the Python Object-Based API by overhauling the generated `__init__` method for `T`-suffixed classes.
Previously, `T` objects had to be instantiated with an empty constructor, and their fields had to be populated manually one by one. This was verbose and not idiomatic Python.
This change modifies the Python code generator (`GenInitialize`) to produce `__init__` methods that are:
1. **Keyword-Argument-Friendly**: The constructor now accepts all table/struct fields as keyword arguments, allowing for concise, single-line object creation.
2. **Fully Typed**: The signature of the `__init__` method is now annotated with Python type hints. This provides immediate benefits for static analysis tools (like Mypy) and IDEs, enabling better autocompletion and type checking.
3. **Correctly Optional**: The generator now correctly wraps types in `Optional[...]` if their default value is `None`. This applies to strings, vectors, and other nullable fields, ensuring strict type safety.
The new approach remains **fully backward-compatible**, as all arguments have default values. Existing code that uses the empty constructor will continue to work without modification.
#### Example of a Generated `__init__`
**Before:**
```python
class KeyValueT(object):
def __init__(self):
self.key = None # type: str
self.value = None # type: str
```
**After:**
```python
class KeyValueT(object):
def __init__(self, key: Optional[str] = None, value: Optional[str] = None):
self.key = key
self.value = value
```
#### Example of User Code
**Before:**
```python
# Old, verbose way
kv = KeyValueT()
kv.key = "instrument"
kv.value = "EUR/USD"
```
**After:**
```python
# New, Pythonic way
kv = KeyValueT(key="instrument", value="EUR/USD")
```
* [Python] Use correct type for str with None
Otherwise mypy will correctly flag code like this
def __init__(self):
self.fooBar = None # type: Optional[str]
error: Incompatible types in assignment (expression has type "None", variable has type "str")
* [Python] Make list type optional as they can contain None
* optionally generate type prefixes and suffixes for python code
* fix codegen error when qualified name is empty
* WIP: Python typing
* more progress towards python typing
* Further iterate on Python generated code typing
* clang-format
* Regenerate code
* add documentation for Python type annotations option
* generate code with Python type annotations
* handle forward references
* clang-format
+/-inf were not being handled, and so invalid typescript was being
generated when a float/double had an infinite default value. NaN was
being handled correctly.
Co-authored-by: Derek Bailey <derekbailey@google.com>
Co-authored-by: Casper <casperneo@uchicago.edu>
* feat: Fixed the issue with nested unions relying on InitFromBuf.
Problem: Issue #7569
Nested Unions were broken with the introduction of parsing buffers with an initial encoding offset.
Fix:
Revert the InitFromBuf method to the previous version and introduction of InitFromPackedBuf that allows
users to read types from packed buffers applying the offset automatically.
Test:
Added in TestNestedUnionTables to test the encoding and decoding ability using a nested table with a
union field.
* fix: Uncommented generate code command
* feat: Added support for fixed sized arrays to python
Problem:
We encountered that using fixed arrays from C++ to python that python would
not read those arrays correctly due to no size information being encoded in the byte
array itself.
Fix:
Encode the sizes within the generated python file during code generation.
Specfically we add GetArrayAsNumpy to the python version of table, which takes as input
the length of the vector. When generating the python message files we include this length
from the VectorType().fixed_length.
* fix: added digit support for camel case to snake case conversion
Problem:
When including a number in the message name we would encounter cases where SnakeCase would
not add the appropirate breaks. e.g. Int32Stamped -> int_32stamped rather than int_32_stamped.
Fix:
To fix this we can add the condition that we check if the current character is not lower and
not a digit, that we check if the previous character was a lower or digit. If it was a lower
or digit then we add the break.
* fix: Array support for structures
Problem:
The python generated code for handling non-struct and struct vectors
and arrays was inconsistent. The calls to populate the obj api was
creating incorrect code.
Solution:
To fix this the VectorOfStruct and VectorOfNonStruct was rewritten
to handle array cases and bring the two methods in line which each
other.
Testing:
PythonTesting.sh now correctly runs and generates the code for
array_test.fbs.
Minor modifications were done on the test to use the new index
accessor for struct arrays and the script correctly sources the
location of the python code.
* chore: clang format changes
* Added code generated by scripts/generate_code. Modified GetArrayOfNonStruct slightly
to allow for function overloading allowing the user to get a single element of an array
or the whole array.
* Added new_line parameter to OffsetPrefix to allow optional new lines to be added.
This allows us to use the GenIndents method that automatically adds new lines instead.
* Reupload of generated code from the scripts/generate_code.py
* Removed new line in GetVectorAsNumpy.
* Updated Array lengths to use Length methods where possible. Added fallthrough for GenTypePointer. Added digit check to CamelToSnake method. Added and modified tests for ToSnakeCase and CamelToSnake.
* Added range check on the getter methods for vector and array types. Renamed == as is for python
* Fix 64-bit default numeric enum values in typescript
If you had a default value that wasn't a valid enum value (e.g., a zero
if you used a bit_flag setting, like you get with AdvancedFeatures
in reflection.fbs), we weren't using BigInt.
* Run generate_code.py
* [DART] Handle deprecated fields & invalid enum defaults
* Update .NET test
* Keep methods with struct name and switch them to default
This PR can help fix the following two issues:
1): A set of simplified API (without struct name) was added in
https://github.com/google/flatbuffers/pull/6336. It causes name
conflict when merging all generated python file into a single one
(the primary usage senario in Google).
2): Flatbuffers 2.0 generates absolute import path, which may cause
name space conflicts. See more details in
https://github.com/google/flatbuffers/issues/5840.
The solution for both is to generate the merged Python code, similar
C++. The merged code will not contain the simplied API, but only the
method with struct name. For issue (1), it will mimic the exactly
usage pattern for Google internal. For issue (2), users can generate
the merged flatbuffer code, without worrying about the imports.
The above idea will be implemented in the following steps:
Step 1 (this PR): revert changes in https://github.com/google/flatbuffers/pull/6336
that set the simplified API as default. Remove statements that the
original API will be deprecated, and reset the original API as default.
Step 2 (the following PR): create a flag to generate the merged code.
The Simplified API will be removed from the merged code, otherwise it
will cause name conflict.
* Update the generated sample code
* Update the generated example code
* Reverst the changes of GetRootAs
* Update examples from grpc/example/generate.sh
* Remove a lot of redundancy from the Python generated code.
Update tutorial to reflect new Python generated code.
* Add aliases for newly deprecated Python generated methods.
This should help with backwards compatibility.
* Fix incorrect names in deprecated comments.
This commit fixes handling of default and NULL `key` fields in `Parser::ParseVector` (#5928).
The JSON generator updated. It outputs `key` fields even if the `--force-defaults` option is inactive.
Additional test cases for `key` added.
* Add vectorNumElements attribute to Builder for simpler vector creation.
This adds a default to EndVector which should simplify its use.
* Update tutorial to reflect new default arg in Python EndVector.
* Remove optional argument to Python EndVector.
* Add generated files.
* Unset Builder.vectorNumElems when not in use.
* Flatbuffers Python Object API
Implement the logic to generate the Python object API that can
unpack the data from a buf class into an object class, and pack
the data of an object class to a buf class.
* Fix the build issues
Remove unused parameters and replace auto in the for-loop statement
with std::string to make it compatible with VS2010.
* Fix the build issues.
* Add support for Array type
Added logic to handle Array type in Python Object API. Updated the
generated code accordingly.
* Fix the old style casting from int to char
* Fixed another conversion from int to char
* Fixed the import for typing
Importing typing may cause errors when a machine do not have the
moduel typing installed. This PR fixes the issue by guarding
"import typing" with the "try/except" statement.
* Fix issue of iterating the vector of import list
* Update the generated examples using generate_code.sh
* Fix the import order for typing
The import list was stored in unordered_set, so that each generated
codes may have different import order. Therefore, it failed in the
consistency test where two generated copies need to have exactly the
same apperance.
* Optimize unpack using numpy
Use numpy to unpack vector whenever it is possible to improve unpack
performance.
Also, added codegen command for Python specificly in generate_code.sh,
because --no-includes cannot be turn on for Python.
* Fix the import order
* Update generate_code.bat for windows accordingly
* Replace error message with pass
Avoid printing error message for every Python2 users about typing.
Replace it with pass.
* Python: Added support for file_identifiers
* Added tests. Fixed file_identifier code.
* Python: Fixed excessive padding of file_identifier. Repaired tests.
* Python: Made code compatible with python2.7
* Python: Typo fix in @endcond
* whitespace normaalization
* Stylistic change from if(not X is None) to if(X is not None). Added comment to type string.
* Python: Added support for automatic code generation of file_identifiers. Added tests for said code generation.
* converted sprintf to snprintf
* Bugfix, added snprint deffinition for MSVC
* changed snprint deffinition for MSVC to sprint_s
* changed scanf to IntToStringHex. Renamed HasFileIdentifier to GenHasFileIdentifier.
* Added updated genereated code to commit
* Python bugix: flatc no longer produces HasFileIdentfier for shcemas with no file identifier
* Added tests to verify `MonsterBufferHasIdentifier` returns false on no Identifier
* Python: added tests for GetBufferIdentifier and BufferHasIdentifier
Python: removed unessasary parenethesis in if statements
Minor format changes.
* Python : correceted instances of keyword arguments being called as positional arguments
* fixed typos and grammer in comments
* Minor style fixes
* Indentation fix
* Equals style changes
* Python: Fixed Alignment Issues. Changed test code to test against atual output
* Ran make(forgot to run make last commit)
* Python: Style changes
* Style changes
* indentation and style
* readded CONTRIBUTING.md
* Formatting tweak
Mostly to make CI run again
* More formatting fixes
* More formatting fixes
* More formatting fixes
* More formatting fixes
* Formatting fix
* More formatting fixes
* Formatting
* ran generate_code.sh
- update C++ monster_test::Color to unsigned type
- update Go Color:ubyte in the go_test.go
- add workaround for unsigned enum in java test
- sync generate.bat and generate.sh
Some generic C++ and Rust code is not generated when unions use type
aliases because of potential ambiguity. Actually check for this
ambiguity and only disable offending code only if it is found.
* added intended use-cases to monster_test.fbs
* added check for `cpp_ptr_type` on hashed fields
added default value 'naked' to `cpp_ptr_type` on hashed fields
* added C++ generation of cpp_type vectors
removed ctor call for vector fields
added condition !vector for cpp_type check
added Pack() and UnPack() code generation for vector of hashes
added generation of correct resolve/rehash for cpp_type elements
* added attribute 'cpp_ptr_type_get' to hold accessor for pointer types possible where '.get()' does not work
use case: cpp_ptr_type:"std::weak_ptr", cpp_ptr_type_get:".lock().get()"
* run flatc to re-generate headers
* added bool param is_ctor to GetDefaultScalarValue() to differentiate between usage places
* modified monster_test.fbs to remove usage of shared_ptr/weak_ptr
reason: STLport does not support std::shared_ptr and std::weak_ptr
* run flatc again to re-generate headers
* fixed symbol unique_ptr not in namespace std when building with STLport
* added support for parsing hash on vector elements
reversed check for scalar to check for vector
added C++ generation of cpp_type vectors
removed ctor call for vector fields
added condition !vector for cpp_type check
added Pack() and UnPack() code generation for vector of hashes
* schema change:
added table Referrable and weak references towards it from Monster
added single_weak_reference to Monster table
changed order with vector_of_weak_references
* re-generated monster schema dependent code
added Referrable.cs to FlatBuffers.Test.csproj
* Fix issue #4389, if field is ommitted the return type is the same for python generator
* Fix issue #4389, bytes is returned when field is omitted
* Fix issue #4389, added generated python code after generator modification
A Namespace object is now guaranteed unique.
This cleaned up some old workarounds and latent bugs.
Change-Id: Ic3f12d89947871b03b2c449ba51b3186f953adde
Tested: on Linux.
Bug: 21336857
* Add numpy accessor to python flatbuffers scalar vectors
* Update python tests to test numpy vector accessor
* Update appveyor CI to run Python tests, save generated code as artifact
* Update example generated python code
* Add numpy info to python usage docs
* Update test schema and python tests w/ multi-byte vector
* did not mean to push profiling code
* adding float64 numpy tests
* Support binary search for struct in cpp
CreateVectorOfSortedStruct is provided for convenience.
* fix continuous-integration error
* add generated files
* compile Ability.cs in csharp test
* compile Ability.cs in csharp
* modify according to code review
A previous commit that added "f" for C++/Java/C# would break
on gcc of constants like 3f, which are now output as 3.0f
Tested: on Linux
Change-Id: If9cabbe3c6d6948a5050b8b123bda9c06e181f52
Implement code generation and self-contained runtime library for Python.
The test suite verifies:
- Correctness of generated Python code by comparing output to that of
the other language ports.
- The exact bytes in the Builder buffer during many scenarios.
- Vtable deduplication correctness.
- Edge cases for table construction, via a fuzzer derived from the Go
implementation.
- All code is simultaneously valid in Python 2.6, 2.7, and 3.4.
The test suite includes benchmarks for:
- Building 'gold' data.
- Parsing 'gold' data.
- Deduplicating vtables.
All tests pass on this author's system for the following Python
implementations:
- CPython 2.6.7
- CPython 2.7.8
- CPython 3.4.2
- PyPy 2.5.0 (CPython 2.7.8 compatible)
