Commit Graph

198 Commits

Author SHA1 Message Date
Derek Bailey
63b7b25289 FlatBuffers 64 for C++ (#7935)
* First working hack of adding 64-bit. Don't judge :)

* Made vector_downward work on 64 bit types

* vector_downward uses size_t, added offset64 to reflection

* cleaned up adding offset64 in parser

* Add C++ testing skeleton for 64-bit

* working test for CreateVector64

* working >2 GiB buffers

* support for large strings

* simplified CreateString<> to just provide the offset type

* generalize CreateVector template

* update test_64.afb due to upstream format change

* Added Vector64 type, which is just an alias for vector ATM

* Switch to Offset64 for Vector64

* Update for reflection bfbs output change

* Starting to add support for vector64 type in C++

* made a generic CreateVector that can handle different offsets and vector types

* Support for 32-vector with 64-addressing

* Vector64 basic builder + tests working

* basic support for json vector64 support

* renamed fields in test_64bit.fbs to better reflect their use

* working C++ vector64 builder

* Apply --annotate-sparse-vector to 64-bit tests

* Enable Vector64 for --annotate-sparse-vectors

* Merged from upstream

* Add `near_string` field for testing 32-bit offsets alongside

* keep track of where the 32-bit and 64-bit regions are for flatbufferbuilder

* move template<> outside class body for GCC

* update run.sh to build and run tests

* basic assertion for adding 64-bit offset at the wrong time

* started to separate `FlatBufferBuilder` into two classes, 1 64-bit aware, the other not

* add test for nested flatbuffer vector64, fix bug in alignment of big vectors

* fixed CreateDirect method by iterating by Offset64 first

* internal refactoring of flatbufferbuilder

* block not supported languages in the parser from using 64-bit

* evolution tests for adding a vector64 field

* conformity tests for adding/removing offset64 attributes

* ensure test is for a big buffer

* add parser error tests for `offset64` and `vector64` attributes

* add missing static that GCC only complains about

* remove stdint-uintn.h header that gets automatically added

* move 64-bit CalculateOffset internal

* fixed return size of EndVector

* various fixes on windows

* add SizeT to vector_downward

* minimze range of size changes in vector and builder

* reworked how tracking if 64-offsets are added

* Add ReturnT to EndVector

* small cleanups

* remove need for second Array definition

* combine IndirectHelpers into one definition

* started support for vector of struct

* Support for 32/64-vectors of structs + Offset64

* small cleanups

* add verification for vector64

* add sized prefix for 64-bit buffers

* add fuzzer for 64-bit

* add example of adding many vectors using a wrapper table

* run the new -bfbs-gen-embed logic on the 64-bit tests

* remove run.sh and fix cmakelist issue

* fixed bazel rules

* fixed some PR comments

* add 64-bit tests to cmakelist
2023-05-09 09:16:30 -07:00
Wouter van Oortmerssen
ed11b08fc9 GenerateText gives text error on failure 2023-05-03 13:03:00 -07:00
Max Burke
a397dd7e8c Optionally generate Python type annotations (#7858)
* optionally generate type prefixes and suffixes for python code

* fix codegen error when qualified name is empty

* WIP: Python typing

* more progress towards python typing

* Further iterate on Python generated code typing

* clang-format

* Regenerate code

* add documentation for Python type annotations option

* generate code with Python type annotations

* handle forward references

* clang-format
2023-04-28 09:38:29 -07:00
Max Burke
56ecc1f548 Optionally generate type prefixes and suffixes for python code (#7857)
* optionally generate type prefixes and suffixes for python code

* fix codegen error when qualified name is empty

* generated code updated
2023-04-25 21:38:16 -07:00
Khanh Nguyen
0916f1c87e Add a FileWriter interface (#7821)
* Add a FileWriter interface

* Change interface

* Provide 2 impl for File interface: FileManager & FileNameManager

* Update

* update

* Update

* Add file_writer file

* Update

* Format files

* Update based on review

* Update

* Format bzl file

* Add LoadFile function

* Format

---------

Co-authored-by: Derek Bailey <derekbailey@google.com>
2023-04-06 01:49:29 +00:00
Ben Beasley
d4d355d883 Fix help output for --java-checkerframework (#7854) 2023-03-09 06:53:37 +00:00
Chuck Atkins
b90cc35a10 Add a --java-package-prefix option to flatc (#7848)
Co-authored-by: Derek Bailey <derekbailey@google.com>
2023-03-02 21:52:03 -08:00
Khanh Nguyen
4a34cd70dc Add Code Generator for idl_gen_fbs to parse .proto files (#7832)
* Add code generator for proto files

* Update

* Add --proto to script

* Remove cmt

* Move proto parsing logic into else block to share same set up logic for code_generator

* Remove IsValidCodeGenerator
2023-03-02 10:01:44 -08:00
Derek Bailey
535ead8d8c [Annotated Buffers] Improve efficiency (#7820)
* AnnotatedBinaryTextGen switch to ofstream instead of building giant string

* Add --annotate-sparse-vectors to reduce AFB size
2023-02-06 21:42:44 -08:00
Saman
f838017860 Parsing from proto should keep field ID. (fixes #7645) (#7655)
* Parsing from proto should keep field ID. (fixes #7645)

* Fix failed tests

* Fix windows warning

* Improve attribute generation in proto to fbs

* Check if id is used twice. fix Some clang-format problems

* Test if fake id can solve the test problem

* Validate proto file in proto -> fbs generation.

* Fix error messages

* Ignore id in union

* Add keep proto id for legacy and check gap flag have been added. Reserved id will be checked.

* Add needed flags

* unit tests

* fix fromat problem. fix comments and error messages.

* clear

* More unit tests

* Fix windows build

* Fix include problems

* Fake commit to invoke rebuild

* Fix buzel build

* Fix some issues

* Fix comments, fix return value and sort for android NDK

* Fix return type

* Break down big function

* Place todo

---------

Co-authored-by: Derek Bailey <derekbailey@google.com>
2023-02-01 11:17:35 -08:00
Khanh Nguyen
08ebd202e2 Final refactor for bfsb_generator* and text generator (#7806)
* Refactor BfbsGenerator to use CodeGenerator interface

* Update

* Refactor bfbs generator

* Refactor bfbs generator for lua and nim. Remove old code that use Generator
interface.

* Update import

* Update CMakeLists

* Update BUILD file

* Update BUILD file for src

* Remove from Android CMakeLists and add error message

* Update

* Add generate root file function to Code Generator interface

* Update

* Update

* Minor format fix
2023-01-31 09:35:34 -08:00
Khanh Nguyen
a105c26eca Refactor usage message (#7803)
* Update usage string formation

* Rework help message to use code generator interface

* update

* refactor
2023-01-28 13:17:36 -06:00
Björn Harrtell
ef76b5ece4 [TS/JS] Entry point per namespace and reworked 1.x compatible single file build (#7510)
* [TS/JS] Entry point per namespace

* Fix handling of outputpath and array_test

* Attempt to fix generate_code

* Fix cwd for ts in generate_code

* Attempt to fixup bazel and some docs

* Add --ts-flat-files to bazel build to get bundle

* Move to DEFAULT_FLATC_TS_ARGS

* Attempt to add esbuild

* Attempt to use npm instead

* Remove futile attempt to add esbuild

* Attempt to as bazel esbuild

* Shuffle

* Upgrade bazel deps

* Revert failed attempts to get bazel working

* Ignore flatc tests for now

* Add esbuild dependency

* `package.json` Include esbuild

* `WORKSPACE` Add fetching esbuild binary

* Update WORKSPACE

* Unfreeze Lockfile

* Update WORKSPACE

* Update BUILD.bazel

* Rework to suggest instead of running external bundler

* Add esbuild generation to test script

* Prelim bundle test

* Run test JavaScriptTest from flatbuffers 1.x

* Deps upgrade

* Clang format fix

* Revert bazel changes

* Fix newline

* Generate with type declarations

* Handle "empty" root namespace

* Adjust tests for typescript_keywords.ts

* Separate test procedure for old node resolution module output

* Fix rel path for root level re-exports

* Bazel support for esbuild-based flatc

Unfortunately, we lose typing information because the new esbuild method
of generating single files does not generate type information.

The method used here is a bit hack-ish because it relies on parsing the
console output of flatc to figure out what to do.

* Try to fix bazel build for when node isn't present on host

* Auto formatting fixes

* Fix missing generated code

Co-authored-by: Derek Bailey <derekbailey@google.com>
Co-authored-by: James Kuszmaul <jabukuszmaul+collab@gmail.com>
2023-01-21 12:22:22 -08:00
Derek Bailey
3b8644d32c Defined CodeGenerator Interface and implement C++ (#7771) 2023-01-08 15:01:33 -08:00
Derek Bailey
641fbe4658 Refactor FlatC to receive FlatCOptions (#7770)
* Refactor FlatC to receive `FlatCOptions`

* switch to c++11 unique_ptr
2023-01-08 13:29:00 -08:00
Chris
c2668fc0e2 Add ts-no-import-ext flag (#7748)
Co-authored-by: Derek Bailey <derekbailey@google.com>
2023-01-07 13:42:28 -08:00
jalitriver
920f3827a0 [C++] Add Command-Line Flag to Suppress MIN and MAX Enums (#7705)
Add the --no-minmax-values flag to prevent flatc from generating C++
enums with MIN and MAX enumerated values that otherwise would be set
to the inclusive lower and upper bound respectively of the enum.

This command-line flag is needed to avoid collisions when an enum that
is being ported to FlatBuffers already has a MIN or MAX enumerated
value.

It is also needed to work around a long-standing problem with
magic_enum that causes magic_enum to not see enumerated values that
are not unique.  For example, if FlatBuffers sets MIN = FOO and MAX =
BAR, MIN and FOO share the same underlying value so they are not
unique.  The same is true of MAX and BAR.  This prevents magic_enum
from converting FOO and BAR to and from strings as well as causing
magic_enum to return a count of enumerated values that is two fewer
than it should be.

Co-authored-by: Paul Serice <paul@serice.net>
2023-01-07 10:33:11 -08:00
Michael Le
e000458bb1 Add --go-module-name flag to support generating Go module compatible code (#7651)
* Add --go-module-name flag to support generating code for go modules

* Rename echo example folder

* Grammar

* Update readme for go-echo example

* Update readme for go-echo example

* Re-enable go modules after test is done
2022-11-22 14:28:01 -08:00
RishabhDeep Singh
879622fc57 Fixes #7345 to add the option to minify enums (#7566)
* Added cpp minified enums

* Update generated files

* remove initializer and fix comma

* Fix .gitignore

* Fix comma

* Add tests for cpp minify enums
2022-11-11 04:47:28 +00:00
Ben Beasley
dbc58ab77c Fix help output for --gen-includes (#7611)
Fixes the --help output documenting the deprecated --gen-includes
option, in which the option name contained a typo (--gen-inclues).
2022-11-08 18:16:17 +00:00
Ben Beasley
2facfeec7e Fix missing spaces in flatc help text (#7612)
* Fix error in --json-nested-bytes help text

Correct “bytesin” to “bytes in”

* Fix missing space in --no-leak-private-annotation help text
2022-11-08 17:59:48 +00:00
Even Rouault
a22434e2a1 Add missing #include <algorithm> for std::min/std::max uses, and #include <limits> for std::numeric_limits<> (#7624) 2022-11-08 09:36:35 -08:00
Paul Harris
f7c511957f Audit and fixups for GCC and Clang (#7212)
Added (for compiler versions that support it):
-Wmissing-declarations
-Wzero-as-null-pointer-constant

Then, fixes to problems identified by the extra warnings
Tested only on GCC 9.4.0

Adjusted the CPP code generator to output nullptr where appropriate,
to satisfy -Wzero-as-null-pointer-constant

Added a lot of 'static' declarations in front of functions,
to satisfy -Wmissing-declarations,
and wrap static function defs in anonymous namespaces.

There are advantages to both anonymous namespaces and static,
it seems that marking a function as static will not publish the name in
the symbol table at all, thus giving the linker less work to do.
2022-08-16 10:48:41 -07:00
Derek Bailey
a66de58af9 Partial support for --ts-flat-files and --gen-all (#7446)
* Partial support for --ts-flat-files and --gen-all

* Add generated code changes

* remove some debugging code left over

* missed grpc files
2022-08-15 16:11:45 -07:00
SF-Zhou
32328075d8 Fix error msg format when generate GRPC failed (#7350) 2022-08-05 22:12:15 -07:00
Derek Bailey
6e2791640e keep-prefix keeps relative pathing (#7394) 2022-07-26 15:11:54 -07:00
mustiikhalil
11a1988705 Started implementation for private flags in rust (#7269)
Adds flag to the cpp to fix bad annotations in all the languages

Addresses comments to the PR, and fixes logic for leaking annotations
2022-06-05 23:04:05 +02:00
mustiikhalil
832c618f5f Adds implementation flag for swift (#7202)
* Adds implementation flag for swift

Forces internal flag when using @_implementationOnly in swift

Fixes access type for verifier functions & encoder functions

Updates generated code

* Addresses PR comments & adds a code gen dir within the swift tests

* Adds test case for no-include

* Fixes code gen script

Removes prefix
2022-04-06 13:31:38 -07:00
Derek Bailey
e2be0c0b06 Handle root offset and root table vtable invalidation (#7177)
* Handle invalid root offset

* Handle vtable offset invalidation

* Added script generator. Add more cases through vtable ref table size

* review responses

* vtable offset validation

* Moved padding insertion to the end. Tests invalid field lenghts

* table offsets validated. Added type after field

* validate string length

* add todo

* invalid vector length

* invalid structs

* general cleanup

* reworded invalid offsets

* example for vector of structs

* invalid vector of tables

* invalid vector of strings

* invalid vector of scalars

* vector of unions

* validate union type value

* invalid vector union type values
2022-03-23 21:51:32 -07:00
Stefan F
824763b316 Typo in flatc options (warning-as-errors instead of warnings-as-errors) (#7180) 2022-03-21 18:08:14 -04:00
Derek Bailey
d3aeee32bb Annotated Flatbuffer Binary (#7174)
* Annotated Flatbuffer Binary

* Various fixes

* Handles old schema

* handle multiple missing fields

* minor edits

* bazel fix, spelling fix, ascii fix
2022-03-18 14:08:05 -07:00
James Kuszmaul
e5f331db99 [TS] Add single-file ts codegen & bazel rule for typescript (#7161)
The headline here is adding a flatbuffer_ts_library rule for generating
typescript code in bazel. This entails some non-trivial other changes,
but ideally none are user-visible.

In particular:
* Added a --ts-flat-file flag that generates a single *_generated.ts
  file instead of separate files for each typescript type. This makes
  bazel much happier.
* Import the bazel rules_nodejs stuff needed to support building
  typescript in bazel
* Move flatbuffers.ts to index.ts because I wasn't sure how to make
  bazel comprehend the "main" attribute of the package.json. Happy
  to take another stab at figuring that out if really needed.
* Fix another couple keyword escaping spots in typescript...
2022-03-10 10:08:13 -08:00
Derek Bailey
914344ea9b some minor help edits 2022-02-22 15:52:51 -08:00
Derek Bailey
70e2f49bff fixed string-json -> strict-json typo 2022-02-21 20:26:40 -08:00
Casper
615616cb55 Change Rust generated file defaults (#7101)
* Change Rust generated file defaults

After #6731, flatc changed its default behavior
for generating rust code to fix some importing issues.
This was a breaking change which invlidated the patch release,
`flatc 2.0.5` (#7081). This PR reverses the default so we can
release a patch update. However, does break Rust users who work at
HEAD.

* Bump flatc patch version (2.0.6)

Co-authored-by: Casper Neo <cneo@google.com>
2022-02-15 11:48:42 -05:00
Derek Bailey
c85fb690f4 Add Parsing Completed function for Rust (#7084) 2022-02-08 07:40:55 -08:00
Derek Bailey
bc366a7f9e Provide a short help text and default in error case (#6992)
* Provide a short help text and default in error case

* clean up short options a bit
2022-01-31 21:24:48 -08:00
Max Burke
1d294a31b8 Implement Serialize on generated rust types (#7022)
* fix for rust build

* Rust: Implement Serialize on generated types

For debugging convenience it is really handy to be able to dump out
types as another format (ie: json). For example, if we are logging a
type to a structured logging system, or even printing it out in a
structured way to the console.

Right now we handle this by shelling out to `flatc` which is not ideal;
by implementing Serialize on the generated types we can use any of the
Serializer-implementing packages for our structured debug output.

* clang-format

* Make the flatbuffers Rust crate only have an optional dependency on the `serde` packages.

* fix warning

* fix rust test build

* Oh yeah this needs to be initialized

* fix toml syntax

* code review feedback

* rebuild test data
2022-01-30 19:29:18 -05:00
Jon Simantov
a2d38fbb98 Add --warnings-as-errors to flatc compiler. (#7034)
* Add --warnings-as-errors to flatc compiler.

With this option set, flatc will return an error on parsing if
any warnings occurred.

* Add unit test for opts.warnings_as_errors.

* Change explicit option setting to default.
2022-01-25 14:01:16 -08:00
Wouter van Oortmerssen
b8aaccee82 Disable parsing of nested_flatbuffers as bytes by default
Parsing as bytes produces buffers that are unsafe to access unless passed thru a verifier,
whereas users could reasonably assume that any JSON parsed without errors is safe to access.
Users that still have legacy JSON files with such bytes in it will get a helpful error point them
to the option to turn on to have it work again.
2021-12-15 10:41:29 -08:00
Wouter van Oortmerssen
f8148b8dad Made FlexBuffers reuse tracker track types 2021-12-14 11:20:23 -08:00
Derek Bailey
3cbc120a08 Refractor Flatc Options (#6987)
* Moved error/warnings to bottom of std error

* Refactor flatc options

* pass program name to flatc

* extra -- in one option

* merge of upstream
2021-12-13 13:56:47 -08:00
Derek Bailey
b78002ff3e Validate C# json/object-api options (#6985) 2021-12-13 11:52:01 -08:00
Wouter van Oortmerssen
e367ca32ad Verifier for FlexBuffers (#6977)
* Verifier for FlexBuffers

* Verifier improvements & fuzzer
2021-12-10 14:59:08 -08:00
lu-wang-g
5fc87f4c4b Enable --gen-onefile in Python (#6953)
* Enable --gen-onefile in Python

Made it possible to generate all python code in one file. Modified
py_test.py so that it can switch between the multi-file code and
the one-file code.

Updated PythonTest.sh and py_test.py so that the multi-file code
and the one-file code can be tested based on the same test code.

* Sync with google/flatbuffers

* Add --gen-onefile to generate_code.py
2021-12-09 15:47:09 -08:00
Derek Bailey
061d61f3f8 Lua Generator using IR. (#6940)
* initial hack to get new Lua generator into flatc

* Starting to output enum defs for Lua

* Continue to work on table generation for Lua

* Finished basic getter access for Lua

* Added ability to get object by index

* Finished struct builder

* aliased reflection to r

* finish table builder generation

* register requiring files

* better generated header info

* Tying up loose ends

* Updated reflection to handle struct padding

* Addd type sizes to reflection

* Fixed some vector indirect issues

* Lua tests passed

* Misc cleanup

* ci fixes 1

* ci fixes 2

* renaming

* up size of type sizes

* manually ran clang-format-11 -i src/idl_parser.cpp

* fixed some windows casting

* remove stupid auto import

* more static_casting

* remove std

* update other build environments

* remove scoped enums

* replaced std::to_string with NumToString

* more win fixes

* more win fixes

* replaced old lua with new

* removed auto import

* review responses

* more style fixes

* refactor bfbs_gen_len to use code +=

* added consts

* fix lambda capture for windows

* remove unused return type
2021-12-02 21:29:19 -08:00
Alec Theriault
c0394bb09c Support --conform for vector of unions (#6883)
It seems like `--conform` already works for vectors of unions - there is
just a spurious check that prevents it from running. Fixes #6882

Also, if schemas do not conform, `flatc` no longer prints out the usage
(since the error is not due to bad usage). Fixes #6496
2021-11-01 11:26:36 -07:00
mustiikhalil
4c7a9c10d3 Adds JSON encoding to swift (#6874)
Updates generated code & removes unneeded protocol

Updates cpp to only generate code when flag is passed

Updates code gen script
2021-10-07 23:22:22 +02:00
Todd Hansen
6f7a57eaa0 [C#] Using 'global::' as qualifying_start_ within BaseGenerator (#6767)
* Update idl_gen_csharp.cpp

Change csharp generator to use "global::" for it's qualifying_start_ to disambiguate namespaces

* regenerate testing files

regenerate testing files

* Missed TableInC.cs

updated with global prefix

* Remove "global::" from qualifying_start_ for csharp generator

* C# global alias

* Tests and docs for --cs-global-alias

Add tests for --cs-global-alias to demonstrate use case for why it's needed.

Add documentation to Compiler.md

* Add also to help text

Add also to help text
2021-09-15 16:57:29 -05:00
Casper
0a3b017f09 Delete IDLOptions::lang (#6841)
Co-authored-by: Casper Neo <cneo@google.com>
2021-09-13 18:18:00 -04:00